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METHOD AND APPARATUS FOR ESTIMATING DELAY AND JITTER 
BETWEEN NETWORK ROUTERS 



Jain, Bijendra N. 
McCloghrie, Keith 

BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates to computer networks, and, more particularly, to 
a method and apparatus for estimating a network performance metric, such as delay 
and jitter, between pairs of routers in a network. 

Description of the Related Art 

As computer networks grow, in terms of the nimiber of routers contained 
therein, the measurement of network performance metrics becomes of increasing 
importance. By measuring such metrics, network parameters can be tuned in order to 
provide optimal performance. Moreover, the network's architecture can be adjusted 
and growth planned to allow the network to grow in a controllable fashion. One such 
metric is the delay experienced by data packets flowing between certain of these 
routers (i.e., travel time between the routers). Another is the jitter, or deviation in 
delay, experienced by such data packets. Thus, there is a growing need to 
continuously monitor network delay and jitter between multiple pairs of routers in a 
network such as an enterprise or service-provider network. In service-provider 
networks, particularly, such performance monitoring is needed in order to verify 
service-level agreements. 

Unfortunately, current methods of monitoring are not as useful as might be 
desired. For example, one current method for monitoring network delay and jitter 
requires the measurement of delay and jitter between every specified pair of routers by 
exchanging probe packets between routers. As will be apparent to one of skill in the 
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art, the number of pairs of routers that need to be monitored in such a scenario grows 
as a quadratic of N, where N is the number of network routers making up the network. 
Thus, such a measurement technique involves calculations on the order of (0(N^)). 

Once generated, the measurement data is collected and processed. The 
5 measurement data can then be made available to other applications, such as for the 
tuning of network parameters and the like. As can be seen from the technique's 
computational order (0(N^)), this measurement scheme does not scale well in large 
networks as the number of specified pairs of routers to be monitored increases 
dramatically. In such cases, the resulting network traffic due to probe packets can be 
10 large and, therefore, unsustainable as a resuh of the bandwidth consumed thereby. 

This problem of scalability may be further compounded by the fact that 
networks that will be deployed in the future will likely be diff-serv (DS) enabled. In 
such cases, delay and jitter characteristics must be monitored for every DS-codepoint 
in use in the network. Diff-serv enabled networks offer a range of data transfer 

1 5 services that are differentiated on the basis of performance experienced by packets 
belonging to a particular aggregated set of applications or flows. An application 
requests a specific level of performance on a packet-by-packet basis, by marking the 
type-of-service (ToS) field in each IP packet with a specific value, also called DS- 
codepoint. This value effectively specifies how an enterprise network or a service 

20 provider network processes and forwards each packet along each hop in the network. 

What is therefore needed is a method and apparatus for the measurement of 
delays encountered by network traffic in traversmg a network, the complexity of 
which preferably grows at a rate less than 0(N^). More preferably, the complexity of 
the measurement scheme should grow at a rate that increases at most linearly with the 
25 number of network routers (designated herein as N) and the number of network links 
(designated herein as M). 
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Summary of the Invention 

The present invention, in various embodiments, addresses tlie aforementioned 
problems by providing techniques for measuring network performance metrics, such 
as delay encountered between nodes in a network. Advantageously, embodiments of 
5 the present invention allow such measurements to be made without generating 
excessive network traffic, and are capable of accounting for delays both between 
nodes (the delay over inter-node links) and within nodes. Also advantageously, 
embodiments of the present invention are less demanding from a measurement 
perspective, being on the order of N+M (0(N+M)). 

1 0 A delay measvirement technique according to an embodiment of the present 

invention is based on the precept, ascertained by the inventors, that a link between 
network nodes will often contribute to the delay encoimtered between several different 
pairs of network nodes. Such a technique identifies the path between each pair of 
nodes by a list of links that form the path. Paths that are orthogonal are treated as 

15 being necessary for describing the delays encountered between nodes. Once delay 

and jitter over the requisite set of orthogonal paths have been measured, delay over all 
other paths can be described in terms of the delay and jitter encountered over one or 
more of these orthogonal paths. Such a technique also lends itself to a vector/matrix 
representation of delay and jitter over the paths, and the use of matrix manipulation 

20 techniques in deriving delay and jitter. 

The present invention allows such measurements to be made in several 
situations. Certain embodiments of the present invention allow delays to be measured 
using a minimal set of paths. This technique can be extended to the case where the 
number of paths used to represent the network being analyzed approaches the number 
25 of paths in the network. Moreover, the techniques presented herein can be extended 
to the case where only a portion of the network's topology is known, for both the case 
of a minimal set of paths and a large number of paths. 

The techniques described herein provide several advantages. These techniques 
address the fimdamental concern of a measurement scheme's scalability in monitoring 
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delay and jitter in increasingly large networks, with known or partly known topology 
and routing. The techniques result in a significant reduction in traffic overhead due to 
exchange of probe packets when compared to existing techniques. The savings can be 
especially significant in networks having a large requirements set. Further, in a 
5 network employing TCP/IP protocols, the methods described herein separately 
accounts for IP-level transfer delay over a link and the delay in processing probe 
packets within a router (i.e., the delay experienced in generating and/or receiving such 
packets), thereby helping to isolate and identify links that are over or under 
provisioned. Simplicity and ease of implementation are also attractive features of a 
1 0 technique according to the present invention. 

In one embodunent of the present invention, a method of determining a 
network performance metric in a network is described. The network includes a 
number of network elements and a number of links. Each of the network elements is 
coupled to at least one other of the network elements by at least one of the links. The 

1 5 method includes measuring a measured network performance metric between a first 
network element and a second network element of a network element pair in a first set 
of network element pairs. The first network element and the second network element 
of the network element pair in the first set of network element pairs is included in the 
network elements. The measured network performance metric is such that a 

20 computed network performance metric between a first network element and a second 
network element of the network elements can be computed using the measured 
network performance metric. 

In one aspect of this embodiment, the computed network performance metric 
is computed. In this aspect, the computed network performance metric is computed 
25 between a first network element and a second network element of a network element 
pair in a second set of network element pairs, using the measured network 
performance metric (as noted above). The first network element and the second 
network element of the network element pair in the second set of network element 
pairs can, for example, be included in the network elements. Moreover, the first set of 
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network element pairs can, for example, be included in a second set of network 
element pairs. 

In another embodiment of the present invention, a method of determining a 
network performance metric in a network is described. The network includes a 
number of network elements. Each one of the network elements is coupled to at least 
one other network element by at least one of a number of links. The method begins 
by identifying pairs of the network elements as being in a first set of network element 
pairs. Next, a first matrix is generated ifrom the first set of network element pairs. 
Each row in the first matrix corresponds to a corresponding network element pair in 
the first set of network element pairs. The first matrix includes independent rows and 
non-independent rows. A second set of network element pairs is then formed. The 
second set of network element pairs contains independent network element pairs in 
the first set of network element pairs. Each one of the independent pairs of network 
element corresponds to one of the independent rows of the first matrix. Next, a 
measured network performance metric is measured between a first network element 
and a second network element of each network element pair in the second set of 
network element pairs. Finally, a computed network performance metric between a 
first network element and a second network element of a remaining network element 
pair in the first set of network element pairs is computed using at least one of the 
measured network performance metrics. The remaining network element pair 
corresponds to a non-independent row of the first matrix. 

The foregoing is a summary and thus contains, by necessity, simplifications, 
generalizations and omissions of detail; consequently, those skilled in the art will 
appreciate that the summary is illustrative only and is not intended to be in any way 
hmiting. As will also be apparent to one of skill in the art, the operations disclosed 
herein may be implemented in a number of ways, and such changes and modifications 
may be made without departing from this invention and its broader aspects. Other 
aspects, inventive features, and advantages of the present invention, as defined solely 
by the claims, will become apparent in the non-limiting detailed description set forth 
below. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention may be better understood, and its numerous objects, 
features, and advantages made apparent to those skilled in the art by referencing the 
accompanying drawings. 

5 Fig. 1 A is a block diagram illustrating a computer system. 

Fig. IB is a block diagram illustrating the various sources of delay 
encountered by network traffic transiting a network. 

Fig. 2 is a block diagram depicting a network in which a delay measurement 
technique according to an embodiment of the present invention can be practiced. 

1 0 Fig. 3 is a flow diagram illustrating a process for computing the round-trip 

delay between pairs of routers according to an embodiment of the present invention. 

Fig. 4 is a flow diagram illustrating a scheme according to an embodiment of 
the present invention capable of processing a requirements set having a large number 
of nodes. 

15 Fig. 5 A is a flow diagram illustrating a method according to one embodiment 

of the present invention for identifying a measurements set such that the resuhing 
matrix has the maximum possible rank. 

Fig. SB is a flow diagram illustrating a method according to one embodiment 
of the present invention for calculating delay and jitter between an arbitrarily specified 
20 pair of routers. 

Fig. 6 is a block diagram illustrating an exemplary enterprise network in which 
a method according to an embodiment of the present invention can be practiced. 

Fig. 7 is a flow diagram illustrating the operations performed in a process of 
forming "clusters" (aggregations of network elements) according to an embodiment of 
25 the present invention. 
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Fig. 8A is a block diagram illustrating the topology of a synthesized network 
and depicting an inferred backbone topology generated by a method according to an 
embodiment of the present invention. 

Fig. 8B is a block diagram illustrating a network and depicting an inferred 
5 backbone topology generated by a method according to an embodiment of the present 
invention. 

Fig. 9 is a flow diagram illustrating a process according to an embodiment of 
the present invention that computes round-trip delay and jitter for router pairs 
contained in a specified requirements set using measurements between router pairs in 
10 the measurements set. 

Fig. 1 OA is a block diagram illustrating an exemplary network in which a 
measurement technique according to an embodiment of the present invention can be 
practiced. 

Fig. 1 OB is a block diagram illustrating an exemplary network after being 
15 apportioned into clusters using a method according to an embodiment of the present 
invention. 

Fig. 1 1 is a flow diagram illustrating the operations performed in the process 
of apportioning delay components according to an embodiment of the present 
invention. 

20 Fig. 12 is a flow diagram illustrating the operations performed in the process 

of apportioning delay components according to an embodiment of the present 
invention, when the size of both a first cluster and a second cluster are greater than or 
equal to two. 

Fig. 13 is a flow diagram illustrating the operations performed in the process 
25 of apportioning delay components according to an embodiment of the present 

invention, when the size of a first cluster is greater than or equal to two and the size of 
a second cluster is equal to one. 

-7- 

#634677vl 

Client Reference: 53382 Weil #1741 



Attorney Docket No. : M-79 1 5 US 



Fig. 14 is a flow diagram illustrating the operations performed in the process 
of apportioning delay components according to an embodiment of the present 
invention, when the size of both a first cluster and a second cluster are equal to one. 

Fig. 15 is a flow diagram illustrating the operations performed in computing 
5 the measurements set for a given requirements set according to an embodiment of the 
present invention, when a maximal set of independent vectors from a corresponding 
set of row vectors is to be found. 

Fig. 16 is a flow diagram illustrating the operations performed in a process of 
computing a measurements set according to an embodiment of the present invention. 

1 0 The use of the same reference symbols in different drawings indicates similar 

or identical items. 

DETAILED DESCRIPTION OF THE INVENTION 

The following is intended to provide a detailed description of an example of 
the invention and should not be taken to be limiting of the invention itself. Rather, 
1 5 any number of variations may fall within the scope of the invention which is defined 
in the claims following the description. 

Introduction 

Embodiments of the measurement techniques described herein address the 
central issue of scalability. These methods allow (a)network performance metrics, 

20 such as delay and jitter, to be measured between a significantly smaller subset of pairs 
of routers, and (b) the corresponding metrics for the remaining specified pairs of 
routers to be computed based on these measurements. These methods include (a) the 
identification of network paths for which measurements must be undertaken, and (b) 
the computation of the metrics for the remaining specified pairs of routers for which 

25 measurements are not taken. 

A definition of the following terms is helpful to the subsequent discussion 
regarding embodiments of the present invention: 
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1 . The "requirements set" is defined herein as the specified collection of pairs of 
routers for which delay and jitter metrics are to be obtained. 

2. The "measurements set" is defined herein as the set of router pairs for which 
delay/jitter measurements are to be taken. A fundamental property of the 

5 measurements set is that, based on measurements thus obtained, one can 

accurately estimate delay/jitter metrics for all pairs of routers specified in the 
requirements set. 

The methods described herein are based on a technique that divides the end- 
router-to-end-router path in terms of processing delay within the source and 

1 0 destination routers and the IP-level transfer delay over intermediate link(s) (and 

router(s), if any) in the network path between the end-routers. Moreover, techniques 
according to embodiments of the present invention can also be used in scenarios in 
which the internal topology of the network being analyzed is unknown. Central to 
these concepts is (a) the identification of a minimal measurements set from a given 

1 5 requirements set and (b) the estimation of delay and jitter for pairs of routers in the 
requirements set. 

The methods described herein are such that only a minimal set of pairs of 
routers need be identified. This minimal set can be constructed such that the set's size 
is no more than N+M (the sum of the number of network routers and links). Even 

20 though the exact size of the measurements set depends upon the topology, routing, 

and the specified requirements set, a method according to the present invention results 
in a considerable savings in network traffic caused by the measurement process. To 
illustrate, in a network having 300 routers and 900 links, if the requirements set 
includes 5000 pairs of routers, the method identifies a "measurement set" consisting 

25 of at most 1200 pairs of routers (a saving of at least 76%). As another example, in a 
network having 200 access-routers connected using a 10-node fully connected mesh 
backbone, the savings can be significantly higher depending upon the specified 
requirements set. A more complete comparison is presented subsequently. 
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The methods described herein can be extended to handle a requirements set 
that includes a significant fraction of all pairs of routers. To cope with such cases, 
the method can be made to directly identify the measurements set and the estimation 
scheme, without having to process each pair of routers specified in the requirements 
5 set. It will be noted that, although the techniques disclosed herein are generally 

applicable to various network protocols and topologies, these techniques are discussed 
herein in terms of TCP/IP protocols. 

Other than resulting in considerably reduced network traffic due to probe 
packets, a monitoring scheme based on this method enables one to estimate the IP- 

1 0 level transfer delay over each router-to-router link in the network. Such an estimate 
factors out the delay in processing (generating/receiving) probe packets within a 
router. This is desirable because the delay in processing probe packets can be 
significant as a resuh of probes being generated and received within the application 
layer. It will be apparent to one skilled in the art that an assessment of IP-level 

1 5 transfer delay over each link assists a network administrator in identifying congested 
links in a functioning network. 

The methods described above are applicable to networks for which (a) 
topology and routing information is available, and (b) changes in topology and routing 
are infrequent. These methods may also be extended to apply to networks that are 
20 built around a network backbone, the topology and routing of which are unknown. 
Such a method directly provides a measurements set and a scheme for estimating 
delay and jitter between the routers on such a network. As a result, one is also able to 
apportion the delay-components between the enterprise network and the backbone 
network. 

25 The techniques described herein provide several advantages. These techniques 

address the fundamental concern of a measurement scheme's scalability in monitoring 
delay and jitter in increasingly large networks, with known or partly known topology 
and routing. The techniques result in a significant reduction in network traffic 
overhead due to exchange of probe packets when compared to existing techniques. 

30 The savings can be especially significant in networks having a large requirements set. 
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Further, the methods described herein separately account for IP-level transfer delay 
over a link and the delay in processing probe packets within a router (i.e., the delay 
experienced in generating and/or receiving such packets), thereby helping to isolate 
and identify links that are over or under provisioned. Simplicity and ease of 
5 implementation are also attractive featiire of these techniques. 

The foilov^ing discussion of measurement schemes according to various 
embodiments of the present invention is divided into four primary sections. First is a 
description of measurement technique that uses a minimal number of router pairs to 
measure delay and jitter in an enterprise network having a known topology. Next, a 

1 0 scheme for taking measurements using a large number of router pairs to measure 

delay and jitter in an enterprise network having a known topology is discussed. The 
estimation of delay and jitter in an enterprise network coupled to a backbone having 
an unknown topology is then discussed. In effect, this situation deals with a portion 
of the network being unknovra (e.g., a service provider backbone) and making the 

15 requisite estimation using a minimal measurements set. This discussion includes 
making such estimations with regard to virtual links between routers within the 
backbone. Finally, estimation of delay and jitter in an enterprise network coupled to a 
backbone having an unknovm topology and using a large number of router pairs is 
examined. 
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I. Measurements Using a Minimal Number of Router Pairs in a Network 
Having a Known Topology 

An active network monitoring tool (referred to generically herein as an 
ANMT) can be used to measure round-trip delay, delay variation (jitter) and packet 
5 loss between a specified set of pairs of routers (or a network switch with a router 
switch module or an attached one-arm router) in a large diff-serv enabled enterprise 
network. By design, ANMT software running as an application on a router (e.g., a 
source router), sends a sequence of user datagram protocol (UDP) packets (also 
known as probe packets) to a specified port on a remote device (e.g., a destination 

1 0 router). Based on information contained in UDP responses received from the 

destination, the ANMT software run on the source computes various statistics, such as 
average round-trip delay, standard deviation of delay jitter, and average fraction of 
packets dropped. An ANMT may also be used to test TCP-level connectivity or to 
estimate one-way packet loss and delay-variation. Additionally, an ANMT allows 

1 5 monitoring of performance between devices in a network which offers differentiated 
services. 

A method accorduig to one embodiment of the present invention permits the 
use of a network measurement tool (e.g., an ANMT) to measure round-trip delay and 
jitter between a much smaller subset of pairs of routers (represented herein by the 

20 notation Q), in order to estimate delay and jitter between pairs of routers in a specified 
set (represented herein by the notation O). This subset Q. includes at most (N+M) 
pairs of routers, where N is the number of enterprise routers, and M is the number of 
links. This is a significant improvement over the complexity of measuring delay 
between every pair of routers in the set O using a tool such as an ANMT. It will be 

25 noted that the number of pairs of routers in O may be as large as N(N+l)/2, or 0(N^). 
Second, the techniques discussed herein recognize that delay encountered in 
generating, receiving and processing probe packets within the transport and 
application layers in a router may be significant. 
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Fig. 1 A depicts a block diagram of a computer system 10 suitable for 
implementing the present invention, and exemplary of one or more of client terminals 
1 12(1 )-(N). Computer system 10 includes a bus 12 which interconnects major 
subsystems of computer system 10 such as a central processor 14, a system memory 
5 1 6 (typically RAM, but which may also include ROM, flash RAM, or the like), an 

input/output controller 1 8, an external audio device such as a speaker system 20 via an 
audio output interface 22, an external device such as a display screen 24 via display 
adapter 26, serial ports 28 and 30, a keyboard 32 (interfaced with a keyboard 
controller 33), a storage interface 34, a floppy disk drive 36 operative to receive a 
1 0 floppy disk 38, and a CD-ROM drive 40 operative to receive a CD-ROM 42. Also 
included are a mouse 46 (or other point-and-click device, coupled to bus 12 via serial 
port 28), a modem 47 (coupled to bus 12 via serial port 30) and a network interface 48 
(coupled directly to bus 12). 

Bus 12 allows data commimication between central processor 14 and system 
1 5 memory 16, which may include both read only memory (ROM) or flash memory 
(neither shown), and random access memory (RAM) (not shown), as previously 
noted. The RAM is generally the main memory into which the operating system and 
application programs are loaded and typically affords at least 16 megabytes of 
memory space. The ROM or flash memory may contain, among other code, the Basic 
20 Input-Output system (BIOS) which controls basic hardware operation such as the 

interaction with peripheral components. Applications resident with computer system 
10 are generally stored on and accessed via a computer readable medium, such as a 
hard disk drive (e.g., fixed disk 44), an optical drive (e.g., CD-ROM drive 40), floppy 
disk unit 36 or other storage medium. Additionally, applications may be in the form 
25 of electronic signals modulated in accordance with the apphcation and data 

communication technology when accessed via network modem 47 or interface 48. 

Storage interface 34, as with the other storage intefaces of computer system 
10, may connect to a standard computer readable medium for storage and/or retrieval 
of information, such as a fixed disk drive 44. Fixed disk drive 44 may be a part of 
30 computer system 10 or may be separate and accessed through other interface systems. 
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Many other devices can be connected such as a mouse 46 connected to bus 12 via 
serial port 28, a modem 47 connected to bus 12 via serial port 30 and a network 
interface 48 connected directly to bus 12. Modem 47 may provide a direct coimection 
to a remote server via a telephone link or to the Internet via an internet service 
5 provider (ISP). Network interface 48 may provide a direct connection to a remote 
server via a direct network link to the Internet via a POP (point of presence). 

Many other devices or subsystems (not shown) may be connected in a similar 
manner. Conversely, it is not necessary for all of the devices shown in Fig. to be 
present to practice the present invention. The devices and subsystems may be 
10 interconnected in different ways from that shown in Fig. 1 A. The operation of a 

computer system such as that shown in Fig. 1 A is readily known in the art and is not 
discussed in detail in this application. Code to implement the present invention may 
be operably disposed or stored in computer-readable storage media such as one or 
more of system memory 16, fixed disk 44, CD-ROM 42, or floppy disk 38. 

1 5 It will be noted that the variable identifier "N" is used in several instances in 

Fig. lA to more simply designate the final element (e.g., servers 1 10(1)-(N) and client 
terminals 1 12(1)-(N)) of a series of related or similar elements (e.g., servers and client 
terminals). The repeated use of such variable identifiers is not meant to imply a 
correlation between the sizes of such series of elements, although such correlation 

20 may exist. The use of such variable identifiers does not require that each series of 
elements has the same number of elements as another series delimited by the same 
variable identifier. Rather, in each instance of use, the variable identified by "N" may 
hold the same or a different value than other instances of the same variable identifier. 

Moreover, regarding the signals described herein, those skilled in the art will 
25 recognize that a signal may be directly transmitted from a first block to a second 
block, or a signal may be modified (e.g., amplified, attenuated, delayed, latched, 
buffered, inverted, filtered or otherwise modified) between the blocks. Although the 
signals of the above described embodiment are characterized as transmitted from one 
block to the next, other embodiments of the present invention may include modified 
30 signals in place of such directly transmitted signals as long as the informational and/or 
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functional aspect of the signal is transmitted between blocks. To some extent, a signal 
input at a second block may be conceptualized as a second signal derived from a first 
signal output from a first block due to physical limitations of the circuitry involved 
(e.g., there will inevitably be some attenuation and delay). Therefore, as used herein, 
5 a second signal derived from a first signal includes the first signal or any 

modifications to the first signal, whether due to circuit limitations or due to passage 
through other circuit elements which do not change the informational and/or final 
fiinctional aspect of the first signal. 

The foregoing described embodiment wherein the different components are 
1 0 contained within different other components (e.g., the various elements shown as 
components of computer system 10). It is to be understood that such depicted 
architectures are merely exemplary, and that in fact many other architectures can be 
implemented which achieve the same fimctionality. In an abstract, but still definite 
sense, any arrangement of components to achieve the same functionality is effectively 
15 "associated" such that the desired functionality is achieved. Hence, any two 

components herein combined to achieve a particular fianctionality can be seen as 
"associated with" each other such that the desired functionality is achieved, 
irrespective of architectures or intermedial components. Likewise, any two 
components so associated can also be viewed as being "operably connected", or 
20 "operably coupled", to each other to achieve the desired functionality. 

Fig. IB depicts a block diagram illustrating the various sources of delay 
encountered by network traffic transiting a network 100. While a method according 
to an embodiment of the present invention may be practiced using any one of a 
number of protocols, Fig. IB is illustrated in terms of a TCP/IP network protocol. 

25 Delay can be apportioned into monitor application response delay (exemplified in Fig. 
IB by monitor application response delays 105 and 110) and network layer transfer 
delays (exemplified in Fig. IB by network layer transfer delays 1 15(1)-(N)) over an 
(end-to-end) network path 120 between a source router 125 and a destination router 
130. In transiting network 100, network path 120 may also transit one or more 

30 intermediate routers (exemplified in Fig. IB by an intermediate router 135). Source 
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router 125, intermediate router 135 and destination router 130 are also designated as 
Rj, R2, and R^, indicating the progression of network traffic transiting network 100. 
In a TCP/IP environment such as that of network 100, monitor application delays 105 
and 110 include, respectively, delays attributable to a network monitor 140, and 
5 transport layer 1 45, a transport layer 1 46 and a network monitor 141. In a similar 
fashion, network layer transfer delays 1 15(1)-(N) include delays attributable to 
moving IP packets through and between IP layers 150, 151 and 152, intermediate 
layers 155, 156 and 157, and physical layers 160, 161 and 162, in the manner shown 
in Fig. IB. As can be seen in Fig. IB, the two principle types of delay components 
1 0 contributing to the round-trip delay between a pair of routers such as source router 
125 and destination router 130 when measTjred using an ANMT can be defined as: 

dj, j = 1, 2, . . ., M, the round-trip IP-level delay in transferring an IP packet 
over a link, Lj, between two adjacent routers; and 

15 

s„ i = 1 , 2, . . ., N, the time to generate, receive and process a probe packet 
within transport and application layers in a router, R^. 

The symbols used herein in describing techniques according to various 
20 embodiments of the present invention for estimating round-trip delay and jitter 
between a specified set of pairs of routers are now presented. 

Fig. 2 is a block diagram depicting a network 200 that illustrates the 
components of the delay existing between various pairs of routers that make up 
network 200. As is illustrated in Fig. 2, network 200 includes a number of routers 

25 (exemplified in Fig. 2 by routers 21 0(l)-(6)) which are coupled to one another by a 
number of interconnecting links (exemplified in Fig. 2 by links 220(1)- (9)). In order 
to facilitate the following discussions, routers 210(I)-(6) are also designated Rj-Rg, 
respectively. In similar fashion, links 220(l)-(9) are referred to using Lj-L,, 
respectively. In the general sense, network 200 can be described as including N 

30 routers, R;, i = 1, 2, . . ., N, and M interconnecting links, Lj, j = 1, 2, . . ., M. A route 
between a given pair of routers, n^, = (R,, Rj), is specified as an ordered list of 
intermediate links that network traffic between R^ and Rj traverses: 

- 16- 
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link_path(R„ Rj) = <Lp, L^, L > (1) 

That is, network traffic moving from source router R, to destination router traverse 
5 the Hnks, Lp, \, and L, in that order, and vice-versa, in the other direction. 

For a given DS codepoint, the routes between routers, R^ and R,, are given in 
Table 1 . It will be noted that the routes in Table 1 are specified for a given DS 
codepoint, and in terms of intermediate links. 





R, 


R2 


R3 


R4 


R5 


R6 






Li 


Li,L2 
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L3, Lg 




R2 






L2 


L4 
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L2, L, 


R3 










u 




R4 














R5 












u 


R6 















10 

Table 1. Routing table for network given in Fig. 2. 

As noted, the two principal types of delay components that contribute to the 
round-trip delay between a pair of routers when measured using an ANMT can be 
defined as dj, j = 1, 2, M (the round-trip IP-level delay in transferring an IP packet 
15 over a link, Lj), and S;, i = 1 , 2, . . N (the time to generate, receive and process a 
probe packet within transport and application layers in a router, Rj). 

As a result, the roimd-trip delay between any given pair of routers, R^ and Rj, is 
given by: 

Delay(R4,Rj) = Si + dp + dq+ ... +d, + Sj (2) 

20 

assuming that the route between R; and R, is known and specified in terms of the 
intermediate links. That is, 

link_path(R„ Rj) = <L^, L^, . . . , L > (3) 

25 
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When used between routers, R, and Rj, a delay monitoring tool such as an ANMT 
measures Delay(Ri, Rj) (which may include measurement error). The requirements 
set and measurements set can now be defined using the framework set forth above. 

The user-specified requirements set of P number of pairs of routers between 
which round-trip delay is to be measured is given by 

0= {7tk,k=l,2, ...,P} (4) 

where 

% = (Ri,R,) (5) 

and so 

a) = {(R„Rj),ij = 1,2, ...,?} (6) 

Normally, the subset will only be a small fraction of ail possible pairs of routers. As 
noted, because roimd-trip delay or jitter is monitored, it is immaterial whether the pair 
of routers is specified as % = (R^, Rj) or as = (Rj, Ri). 

The estimate is based on measiirements between a subset of pairs of routers 
(and indeed what can be a minimal subset of router pairs), which may be referred to 
by Q. That is, for every (R^, Rj,) e O, the delay between the pair of routers (Rj, Rj) 
can be computed (or estimated) in terms of measured delay between one or more pairs 
of routers {(Rp, Rq)} e Q. The set, Q, is also referred to herein as the measurements 
set. A method according to the present invention identifies the measurements set, Q, 
for a given requirements set, <1>. 

Determination of the Measurements Set 

A method according to an embodiment of the present invention is now 
described that is capable of identifying the collection of pairs of routers, or Q, such 
that the delay between a given pair of routers (R^, Rj,) g O can be computed from 
measurements between a subset of pairs of routers (Rp, Rq,) e Q.. 
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If the number of pairs of routers in O is defined as P (or, P = |0|), the round- 
trip delay between the pair of routers, (R^, R,,) e (D is denoted by = Delay(Ri, Rj). 
In view of the definition of delay components, it can be seen that, for every k = 1 , 2, 
...,P: 

5 

z, = Delay(Rj, R^) = s^ + dp + d^ + ... + d,+ Sj (7a) 
assuming that the route between R^ and Rj is knovm and given by 
1 0 Hnk_path(Ri, R^) = <Lp, L,, . . . , L > (7b) 

Using a vector notation, z,, = Delay(Ri, Rj) may be re-written as 

Zk = Delay(Ri, R^) (8a) 

15 

= Si + dp + dq+ ... +d, + Sj (8b) 
= FkX (8c) 
And further, the set of Zj, (k = 1 , . . . , P) can be re-written 

20 

Z= [Zi Z2 ... Zpf 

is a column vector of size P. Further, 
25 Fi^=[0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0] (9) 

is a row vector of size N+M, and the 1 's in F,, appear in columns p, q, . . . , r, and in 
columns M+i and M+j . 

The ordered collection of delay components can be re-written in the form of a 
30 vector of size N+M (the delay-components vector). 

X=[di,d2, ...,dM,Si,S2, ...,SHf (10) 



[d, ] (11) 
[d^ ] 
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[s, 

[ - 

[Sn 



As noted, the round-trip delay between the specified pairs of routers in O = 
= (Rj, Rj), k= 1,2,..., P}, where Zk = FkX andk= 1, 2, P, can be re-written as a 
10 vector equation 

z = Fx (13) 



z= [zi ], F= [¥, ] (14) 
[Z2 ] [F2 ] 

[... ] [- ] 

[zp ] [Fp ] 

20 

The P X (N+M) matrix, F, plays an important role in determining the subset of pairs of 
routers between which delay measurements are necessary. 

Consider the row vectors, F^, k = 1 , 2, . . ., P and let Q be the maximum 
number of independent row vectors. As can be seen, Q is equal to the rank of matrix 
25 F (i.e., Rank(F)). Without loss of generality, Fi Fj . . . Fq be the independent row 

vectors of matrix F. This is without loss of generality because the row vectors F^ in F 
(and similarly the corresponding pairs of routers in O) can be re-arranged, if 
necessary. Then, every row vector F,,, k = Q+1, Q+2, . . ., P, can be expressed as a 
linear combination of row vectors, F^, k = 1, 2, . . ., Q. 

30 The relationship between the linearly independent and linearly dependent rows 

of F is now examined. If F„ F2, . . . , Fq is a maximal set of linearly independent rows 
of F, then row vectors Fq+j, Fq^.2, . . . , Fp can be expressed as a linear combination of Fj, 
F2, . . ., Fq. In other words, F^ can be expressed in terms of: 
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Fk = Ei^i Q (a^, Fi), k = Q+1, Q+2, P (15) 

The constants ^ may be re-organized in the form of a row vector (of size Q) in the 
following manner: 

5 

ak= Ki, ak,2, ak,Q], k- Q+1, Q+2, P (16) 

The values of may be aggregated to form the PxQ matrix, A, which can be defined 
as: 

10 

A= [I ] (17) 

[Wq+i ] 
[CCq+z ] 

[... ] 

15 [ttp ] 

where the matrix, I, is a Q x Q matrix. Because delay (zj is a random variable, delay- 
jitter between pair of routers (% = (R;, R^)) can, be defined as follows: 

20 Delay- Jitter(R„ R^) = V E{ (z, - E{z^}f } (18) 

where E{.} is the expectation operation. 
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In regard to a measxarement method according to certain embodiments of the 
present invention, certain assumptions concerning the network being analyzed are 
now summarized. The network is assumed to have the following characteristics: 

5 • For the embodiments discussed above, it is assumed that the network's topology is 
known. It will be noted that, because a route between a pair of routers is specified 
in terms of links, there is no constraint on the existence of parallel links between a 
pair of enterprise routers. 

• Further, it is preferable that the topology not change frequently. If the topology of 
10 the network changes, then the procedure described herein should be carried out 

afresh. 

• For each DS codepoint, the route between every pair of enterprise routers should 
be known. 

• The route between every pair of enterprise routers for a given DS codepoint is 
15 symmetric. 

• The delay and jitter characteristics between a pair of routers may be different in 
the two directions. 

In other words, certain of the procedures described herein are preferably used 
20 together with techniques that monitor changes in topology and in routing in measuring 
delay and jitter in the network. Once a change in topology or in routing is detected by 
these techniques, the method according to an embodiment of the present invention 
should be performed once again. 
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As a consequence of the foregoing manipulations, certain results can be 
deduced. As outlined below, these results include: 

1 . the fact that, for every (R^, Rj) g Delay(R„ Rj) can be expressed as a linear 
combination of one or more Delay(Rp, Rq), (Rp, Rq) e Q; and 

5 2. the fact that the number of router-to-router delay measurements required to 

estimate delay between an arbitrarily specified subset of pairs of routers is at 
most M+N. 

These results are set forth in detail below. 

First Result 

10 The first result is determined as follows. Consider the matrix F = [F, . . . 

Fp]^, corresponding to the given set of router pairs in the requirements set, O. Then, 
the minimal subset of pairs of routers, Q, between which delay must be measured is 
given by the collection of all pairs of routers, (Rp, Rq), that correspond to the maximal 
set of independent rows of F, with respect to {Ft, k = 1 , 2, . . . , Q } . In other words, the 

15 measurements set can be represented as: 

Q = {(Rp, Rq), such that Delay(Rp, R,) = F^ x, k = 1 , 2, . . . , Q} (19) 

As a consequence, for every (R^, Rj) e <!>, Delay(Ri, R|) can be expressed as a linear 
20 combination of one or more Delay(Rp, Rq), where (R^, Rq) e Q. 

The above result is illustrated by the following example. Consider the 
network of Fig. 2, where N=6, M=9. If the requirements set, 0) = {(R3, R4) (R3, R5) 
(R4 Rg) (R5, Rg)}, then from routing table given in Table 1 for a given DS codepoint: 

25 Zi = Delay(R3, R4) = S3 + dg + dg + S4 

Z2 = Delay(R3, R5) = S3 + dg + S5 
Z3 = Delay(R4, R^) = S4 + dg + dg + Sg 
Z4= Delay(R5, Rg) = Sj + dg + Sg 

30 That is, 
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Z=[ZiZ2Z3Z4f = Fx (20) 

where 

x = [di, dj, ds, Si, S2, Sg]^ (21) 
The F matrix is a 4x15 matrix of the form: 
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Z4 
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1 


1 



It is verified subsequently herein that Rank(F) = 3, and that Zj, Z3 and Z4 are 
independent row vectors, and that Zj = Zj + Z3 - Z4. Thus, the measurements set, Q = 

{(R3,R5),(R4R,),(R5, R^)}- 

Using a different requirements set for network 200, if 

15 

O = {(R„ R2), (R2, R3), (R„ R4), (R2, R4), (R2, R5), 

(R3, R5), (R3, Re), (R4, R5), (R5, R.), (Ri, R3), 

(R2, K), (R„ R5), (R4, Rfi), (R„ R*)} (22) 
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the resulting matrix F is 
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It can be shown that the matrix F has rank 13, and that the first 13 rows of F 
5 are independent. Further, Z14 = Z12 + z^ - Zg. Thus Q = {(Ri, Rj), (R2, R3), (Rj, R4), (R2, 
R4), (R^, R5), (R3, R5), (R3, Re), (R4, R5), (R5, Re), (Ri, R3), (R2, Re), (Ri, R5), (R4, Re)}- 
It will be noted that Q. is also the subset of measurements required to estimate the 
delay between an arbitrary pair of routers. The second result is now examined. 

Second Result 

1 0 The minimal subset of pairs of routers, between which delay should be 

measured, so that delay between every other pair of routers can be obtained, can also 
be determined . The number of router-to-router delay measurements required to 
estimate delay between an arbitrarily specified subset of pairs of routers is at most 
M+N. In other words: 

15 

Q = |Q| = Rank(F) < min(|<D|, N+M) (23) 

This is so, because the matrix, F, is a |a>| X (M+N), the rank of which, with respect to 
Q, is at most |<I>| or (M+N), whichever is smaller. As is evident from the approximate 
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minimum savings in network traffic set forth in Table 2 below, the advantages 
provided by the foregoing results can be substantial. 
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5 

Table 2. Approximate minimum saving in terms probe packet traffic. 

The theoretical underpinnings of the foregoing results are now examined. 
Theoretical Underpinnings Of The Measurement Scheme 

10 A measurement scheme according to certain embodiments of the present 

invention, as described herein, makes Q measurements between the collection of Q 
pairs of routers, specified by Q. It will be noted that Q=|0|, and that Q. is the 
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collection of pairs of routers, (Rp, Rq), that correspond to the maximal set of 
independent rows of F, with respect to {F^, k = 1, 2, . . ., Q} (i.e., the measurements 
set). 

Network analysis agents (e.g., ANMTs) can be executed, for example, as 
5 transport applications in two routers, and Rj, to measure round-trip delay and jitter 
between R; and Rj. Such a measurement, denoted y^ yields: 

yk = Delay(Ri, Rj) + Vk = S; + dp + dq + ... + d„ + Sj + (24) 

10 It is assumed that the intermediate links between the two routers are Lp, Lq, . . ., 

L„ that the measurement error is v^. As noted, processing delays encountered by 
probe packets within source and destination routers, R^ and Rj, respectively, are 
defined as Sj and Sj, and dp, dq, d^ are round-trip IP-layer transfer delays 
encountered by probe packets over intermediate links along the route 

15 

link_route(Ri, R^) = <Lp, Lq, . . . , L > (25) 

The k* measurement, yt, k = 1,2, . . ., Q, of round-trip delay between the k"" pair of 
routers, R^ and Rj, in Q, may be re-written as: 

20 

y^ = Delay(Ri, Rj) + Vk = S; + dp + dq + ... +d, + Sj + v^ (26) 

= Fk X + Vk 

25 where the row vector (of size M+N) 

Fk = [0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0] (27) 
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5 



and the 1 's in Fj, appear in columns p, q, .. ., r, and in columns M+i and M+j. The 
following can be defined: 

y-[y.y2-yQf (28a) 

x= [di, dj, dw, Si, S2, s^f (28b) 
v=[v,V2... Vq^ (28c) 

where vector y, of size Q, represents the collection of measurements, vector x, of size 
M+N, is the collection of delay components and vector v, of size Q, is the 
measurement error in y. The measurements, y,,, k = 1, 2, . . ., Q, taken together, may 
now be re-written in the form of a vector equation 

y = H X + V (29) 



where the Q x (M+N) matrix H = [Fj^ . . . Fq^]^. Because the k meastirements 
correspond to pairs of routers identified by Q, the row vectors, F^, k = 1, 2, . . ., Q, are 
all independent. As a consequence, matrix H has full rank with respect to Q. 

20 Consider, for example, the network given in Fig. 2, where N=6 and M==9. The 

routes between pairs of routers, R, and Rj, are given in Table 1. Let Q = {(R3, R5) (R4 
R^) (R5, Rg)} be the specified subset of pairs between which delay is measixred. From 
Table 1, 

25 yi = Delay(R3, R5) + Vi = S3 + d^ + S5 + Vi (30a) 

y2 = Delay(R4, R^) + Vj = S4 + dg + i + s^ + Vj (30b) 
y3 = Delay(R5, Rg) + V3 = S5 + d^ + s^ + V3 (30c) 



That is, 



(31) 
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y = [Yi y2 

X = [di, dj, dg, Si, S2, s^Y 

V = [Vi V2 V3]^ 

and 3x15 matrix H is: 



(32a) 
(32b) 
(32c) 
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Calculating Delay Within A Requirements Set Using A Measurements Set 

In this section, a methodology for computing the round-trip delay between 
every pair of routers within a given requirements set, (Rj, Rj) e O, based on 
measurements between pairs of routers identified by the measurements set, Q (a 
subset of the requirements set), is described. This approach is particularly useful 
when measurement errors are negligible. 

As previously noted, the delay vector (z) corresponding to the requirements set 
is given by 



z = F X 



(33) 



while the measurement vector (y) corresponding to the measurements set is given by 
25 y = H X (34) 

where 



H = [F/F/...FqY 



(35) 
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Because {Fi, Fj, . . . , Fq} is the maximal set of independent rows of F, every row 
vector Fk can be expressed as a linear combination of Fj, Fj, . . . Fq. 



As a result 



z=Fx=AHx 



(36a) 



where 

10 A=[IaY (36b) 



and the row vectors a^, k = Q+1, Q+2, ... P are used to express F^ in terms of Fj, Fj, 
.. ., Fq (as discussed above in equation (17)). 

1 5 A General Procedure For Employing A Measurements Set In Determining Delay For 
A Requirements Set 

A procedure to deterministically compute the round-trip delay between pairs 
of routers, given by O, from measured round-trip delay between the Q number of 
pairs of routers, identified by Q, can be defined using the preceding results. As can be 
2 20 seen, the computed round-trip delay. A, between the P pairs of routers in O is given by 

A = A y (37) 



where y^, k = 1, Q, are delay measurements between the Q pairs of routers in Q, 
25 and Aj, j = 1, 2, . . . , P, are delay between the P pairs of routers in <1> . 

Fig. 3 illustrates a process of deterministically computing the rovmd-trip delay 
between pairs of routers. The process begins with the identification of the 
requirement set (step 300). The F matrix is then determined from the given 
requirements set (step 305). Once the F matrix has been derived, the rank of F is 
30 computed (step 3 1 0). The rank of F indicates the number of independent rows 
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contained in F. The rank of F may be represented, for example, by Q. Once the rank 
of F has been computed, a maximal set of independent rows of F is identified (step 
315). If the first Q rows of F are not independent (step 320), the rows of F (as well as 
router pairs in the requirement set) are reordered such that the first subset of Q row 
5 (i.e., F], F2, Fq) are independent (step 325). 

Once the rows of F (and router pairs in the requirements set) have been 
reordered, or if such reordering is unnecessary (i.e., the first Q rows of F are 
independent) (step 320), the first Q pairs of routers in O (router pairs in the 
requirements set) are copied to form the subset of Q, also referred to herein as the 

10 measurements set (step 330). Once the measurement set Q (or equivalently, the set of 
independent vectors Fl, F2, . . FQ) have been identified, row vectors of a,,, are 
derived such that F,, can be expressed as Fj,= a,,iFi + 0.^2^ ■ ■ • '^^kc^Q- This is done 
for each value of k = Q+1, Q+2, . . ., P (step 335). From these vectors, the A matrix is 
constructed (step 340). A network monitoring tool is then used to measure round-trip 

1 5 delay between pairs of routers identified by Q, and the measurements are organized in 
the form of a column vector, y, of size Q (step 345). The round-trip delay can then be 
computed using the vectors thus derived (step 350). The above procedure may be 
summarized as follows: 
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BEGIN 

Input requirements set O; 

Determine F matrix from the given requirements set, O, and the routing table; 
Q = Rank(F); 

5 Identify a maximal set of independent rows of F; 

If necessary, reorder the rows of F (as well as pairs of routers in O), such that 
the first subset of Q rows, Fj F2 ... Fq, are independent; 

Copy the first Q number of pairs of routers in 5> to form the subset Q; 

10 

Determine vector a,,, k = Q+1, - . ., P, such that Fj, can be expressed in terms of 

Fi,F„...,Fq; 
Construct the A matrix; 

1 5 Measure round-trip delay between router pairs in Q, and organize them as y . 

Compute round-trip delay using the equation A = Ay. 

END 

It will be noted that a negligible measurement error is preferable. It will also be noted 
20 that only a subset of the non-independent rows of F (i.e., F;,, where k is one (or more) 
of Q+1, P) may be of interest, in which case only a subset of the independent rows 
of F may be needed to describe such Fj,. 

Each of the blocks of the flow diagram of Fig. 3, and those depicted in 
subsequent figures, may be executed by a module (e.g., a software module) or a 

25 portion of a module or a computer system user. The methods described herein, the 
operations thereof and modules for performing such methods may therefore be 
executed on a computer system configured to execute the operations of the method 
and/or may be executed from computer-readable media. The method may be 
embodied in a machine-readable and/or computer-readable medium for configuring a 

30 computer system to execute the method. The software modules may be stored within 
and/or transmitted to a computer system memory to configure the computer system to 
perform the functions of the module. Alternatively, such actions may be embodied in 
the structure of circuitry that implements such functionality, such as the micro-code of 
a complex instruction set computer (CISC), firmware programmed into programmable 

35 or erasable/programmable devices, the configuration of a field-programmable gate 
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array (FPGA), the design of a gate array or fuil-custom application-specific integrated 
circuit (ASIC), or the like. 

Those skilled in the art will also recognize that the boundaries between 
modules and operations depicted herein are merely illustrative and alternative 
5 embodiments may merge such modules or operations, or impose an alternative 
decomposition of functionality thereon. For example, the actions discussed herein 
may be decomposed into sub-operations to be executed as multiple computer 
processes. Moreover, alternative embodiments may combine multiple instances of a 
particular operation or sub-operation. Furthermore, those skilled in the art will 

10 recognize that the operations described in exemplary embodiment are for illustration 
only. Operations may be combined or the functionality of the operations may be 
distributed in additional operations in accordance with the invention. As will also be 
apparent to those of skill in the art, methods for determining delay and jitter described 
herein may employ other techniques (similar in effect to those described herein) to 

15 make such determinations, and such alternative techniques are intended to be 
comprehended by the methods and apparati discussed herein. 

As an example of the operation of such a process, the network of Fig. 2 can 
again be considered, where N=6, M=9. The routes between pairs of routers, and Rj, 
are given in Table 1. Let cD - {(R3, R4) (R3, R5) (R4, R^) (R5, K)}- As before 

20 



z = F X 



(38) 



where 



25 



Z = [Zj Z2 Z3Z4]^ 



(39) 



x=[di, dj, 



dg, Sj, S2, Sf^Y 



(40) 
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and the F matrix is a 4x1 5 matrix of the form: 





D, 


D2 


D3 


D4 


D5 


D5 


D7 




D9 


s, 


S2 


S3 


S4 


S5 


S6 














1 




1 








1 


1 


















1 












1 




1 




Z3 
















1 


1 








1 




1 


Z4 


















1 










1 


1 



Because Rank(F) = 3, and Zj, Z3 and Z4 are independent row vectors, the row vectors, 
5 Fk, can be re-ordered as those corresponding to Zj, Z3, Z4, z,. The re-ordered F matrix 
appears as: 





D, 


D2 


D3 


D4 


D5 




D7 


Ds 


D9 


Si 




S3 


S4 


S5 


S5 














1 












1 




1 




Z3 
















1 


1 








1 




1 


Z4 


















1 










1 


1 














1 




1 








1 









The requirements set <I> is also reordered as {(R3, R5) (R4, R^) (R5, R6)(R3, R4)}- 
10 Further, Q = {(R3, R5), (R4 R^), (R5, R<i)}. The resulting H matrix appears as: 





Di 


D2 


D3 


D4 


D5 




D7 


Ds 


D9 


Si 




S3 


S4 


S5 


§6 














1 












1 




1 




















1 


1 








1 




1 


74 


















1 










1 


1 



Now, because Zj = + Z3 - Z4, the matrix a may be computed: 





F2 


F3 


F4 


Fi 


1 


1 
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As a consequence, the A matrix appears as: 





F2 


F3 


F4 




1 






F3 




1 




F4 








Fi 


1 


1 


-1 



The computed estimate of delay z = [Zj Z3 Z4 z^f is given by [Aj A3 A4 A{f = A y = A 
5 [y^YsyJ"- 

A Formalized Description Of A Method For Calculating Delay And Jitter Within A 
Requirements Set Using A Measurements Set 

A procedure according to a method of the present invention that computes the 
round-trip delay and delay-jitter between every pair of routers in the requirements set 
10 (using every router pair = (Rj, Rj) e <1>) using measurements between pairs of 

routers identified by the measurements set, Q, is now described using a formal syntax. 
The scheme is particularly useM when the measurement errors are small and no a 
priori delay measurements are available. 



15 

BEGIN 

/* The following procedure assumes that the topology and routing table, */ 
/* for a given DS codepoint, is known. Where necessary, this procedure */ 
/* is executed for each DS codepoint under consideration. */ 

20 

/* If the network topology and/or the routing changes, this procedure */ 
/* should be executed afresh with the new topology or new routing table. */ 

/* Specify the collection of pairs of routers */ 
25 0 = {% = (Ri,Rj),k=I,2, ...,P}; 

/* Construct the delay-components vector */ 
x=[di,d2, ...,d^, Si, S2, ...,s^Y; 

30 

For each k= 1,2, ...,P,do { 

Identify the pair of routers % = (R;, R,), n^^ g 
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Obtain the route link_path(R„ Rj) = <Lp, L^, . . ., L >; 
Determine the row vector, F^, such that = Delay(Ri, Rj) = Fj, x 

}; 

Construct the P x (N+M) matrix, F; 
Identify a maximal set of independent rows of F; 
Q = the number of maximal set of independent rows of F; 
If necessary, reorder rows of F and pairs of routers in O such that the first Q 
rows, Fj, Fj, . . ., Fq, are independent; 

For eachk = Q+l,Q+2, ...,P, do { 

Determine a^j, ak2, • • ., otkQ, such that Fj. = 2i=i,...,Q (otk,; F;); 
Construct the row vector = [a,, i, 2, ■ • ■ , cXj. q] 

}; 

Copy the first Q pairs of routers from O to form the subset Q; 
Identify the start time to, and the measurement interval, dt; 

to = start_time; /* start_time will most likely be 0 (start at time 0) */ 

While (Ifmished) { 

For eachk= 1,2, Q, do { 

During the time interval, (tg = to+dt), use a network 

management tool (e.g., ANMT) to measure average 
round-trip delay, y^, and delay-jitter, y^, between the pair 
of routers, % = (R^, Rj) 

}; 

/* Estimate round-trip delay vector, A^, for each pair of routers in cj) */ 
For each router pair inO { 

For each k- 1, 2, Q, do { 

Atc = yk 
}; 

For each k = Q+1, Q+2, P, do { 

Ak = 2:i.i,...,Q (ttk^iyi) 
}; 

} 

/* Compute round -trip delay -jitter, <y^, for each pair of routers in O */ 
For each router pair in O { 

Foreachk= 1,2, ...,Q,do { 
cJk = Yk 

}; 

For each k = Q+1, Q+2, P, do { 
-36- 
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to = to+dt 

} 



As noted, the above scheme is appHcable where no other delay measurements or a 
priori estimates are available and assumes that the measurement error is negligible. 

As an example, network 200 may again be considered (network 200 has N=6, 
M=9, as noted). The routes between all pairs of routers, and Rj, are given in Table 
1 . Using the technique just described, the network can be analyzed as follows. Let (1> 
= {Til = (R3, R4), = (R3, Rj), = (R4 Rg), 7C4 = (R5, R^)}. The delay-component 
vector, X = [d], dj, . . dg, Sj, Sj, . . ., Sg]^. Because 



Zi = Delay(R3, R4) = S3 + d6+ dg + S4 
Z2 = Delay(R3, R5) = S3 + d^ + S5 
Z3 = Delay(R4, R^) = S4 + + d^ + 
Z4= DelayCRj, R^) = s,- + dg + Sg 



(41a) 
(41b) 
(41c) 
(41d) 



the delay vector, z = [zj Z2 Z3 Z4]^, may be written as z = F x, where the 4x1 5 matrix F 
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Because Zj, Z3 and Z4 are independent row vectors, Q = 3. The row vectors, F^, are re- 
ordered as those corresponding to Zj, Z3, Z4, Zj. The re-ordered F is: 



Because Zj = + Zj - Z4, the vector a, = [1 1 -1]. As a result, ^^={7:2 = (R3, R5), = 
(R4 Rg), 714 = Re)}' with the corresponding H matrix of the form given in the 
previous example. Next, to is set to 0, and dt is set to 60 (seconds), for example. 

During the interval (to, tg+dt), a network probe tool is used to measure round- 
trip delay and jitter between pairs of routers, {tIj = (R3, R5), = (R4 Re), ^4 = Re)} 
to yield delay measurements, yj, yj, and y4, and delay-jitter measvirements, 72, 73, and 
Y4. An estimate of delay between pairs of routers in O is given by 



A2 = y2 
A3 = y3 

A4 = y4 

Ai = y2 + ys - y4 



(42a) 
(42b) 
(42c) 
(42d) 



An estimate of delay-jitter between pairs of routers in O is given by 



^2 = 12 (43a) 

as = 73 (43b) 

CT4-y4 (43c) 

a, = V (72^ + 73^ + 74^) (43d) 

25 Measurement and estimation can then be repeated with to = to+dt. 
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II. Measurements For a Large Number of Router Pairs in a Network With 
Known Topology 

A method for estimating delay, including packet transfer delays and ANMT 
processing delays, for a requirements set containing a large number of nodes 
5 according to certain embodiments of the present invention is now described. Such a 
method accounts for the delay encountered in transferring information over links in a 
network, and also accounts for delays within the end-point nodes, and intermediate 
nodes along the path between the end-point nodes. Estimation of the vector x, 
includes IP-level transfer delay, dj, over each link, Lj, and of processing delay caused 

10 by the network monitoring tool(s) employed within each router (defined as Sj) may 
be performed used a method described below. The need to estimate the IP-level delay 
over each link stems firom the fact that a measurement between the pair of routers 
corresponding to a link, made using a network monitoring tool, generally include the 
time required to generate, receive and process probe packets within the pair of source 

15 and destination routers. To that end, an estimate of delay incurred due to protocol 

processing (referred to generically herein as IP-level transfer delay) over a link better 
reflects the delay that a typical packet (referred to generically herein as an IP packet) 
encounters over each hop. 

If an estimate of delay due to processing within each router is additionally 
20 available, then one may estimate the delay experienced by probe packets between any 
pair of routers. In other words, once an estimate of vector x is available, an estimate of 
Delay(Rp, R,) may be obtained, because Delay(Rp, Rg) = x for any specified pair of 
routers, (Rp, R,). 
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The Measurements Set, Q 

To review, the measurements set, Q, corresponding to all pairs of routers may 
be defined as the minimal subset of pairs of routers, between which delay should be 
measured so that delay between every pair of routers in the network can be obtained. 
5 The number of router-to-router delay measurements required to estimate delay 

between an arbitrarily specified subset of pairs of routers is at most M+N. As noted: 

Q = |Q| = Rank(F) < min(|<D|, N+M) (45) 

10 This is so, because the matrix, F, is a |0| X (M+N) matrix, the rank of which, with 
respect to Q, is at most the lesser of |^| or (M+N). 

The approach outlined previously may be used to obtain the subset of pairs of 
routers, fi, between vs?hich delay and jitter measurements are made, corresponding to 
the set of all pairs of routers, 

15 

0)= {(Rj,R3),i = 1,2, ...,N,j = l,2, ...,N,i<j } (46) 

Such an approach requires that Delay(Rj, R,) be expressed as Fj, x, for each pair 
of routers, k = 1 , 2, . . ., N*(N-l)/2, and the maximal subset of linearly independent 
20 row vectors, Fj, be identified by determining the maximal set of independent set of 
row vectors. As can be seen, this can be a computationally complex procedure 
because the procedure involves vectors of order N^ (0(N^)). 

Alternatively, one may directly identify the subset Q. A method according to 
one embodiment of the present invention that identifies a smallest subset of pairs of 
25 routers, such that the delay between every other pair of routers can be expressed as a 
linear combination of delay between the pairs of routers in subset, Q, is described 
below. The size of the matrix, Q, is no more than N+M. Further, the corresponding H 
= [Hi^ H/ ... Hq^\ is of fiill rank, Q. 
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It will be noted that the size of Q (i.e., Q) can be in fact smaller than N+M, if 
for some router, R^, there does not exist a path between two routers, Rp and R^, such 
that the route between R^, and R^, passes through Rj. That is, for some Rj, 

5 router_path(Rp, R^) ^ <Rp, . . . , R,, . . . Rq> for all Rp 9^ r. and R^ ?i R; (47) 

The immediate consequence of this observation is that it would not be possible to 
isolate, and thus separately estimate, the delay within router R, due to processing 
required by the network monitoring software, except in combination with delays over 
10 links connecting R; to its neighboring routers. The details of such a technique are now 
discussed. 

A Measurement Scheme for Estimating Delay Components 

A scheme according to an embodiment of the present invention capable of 
processing a requirements set having a large number of nodes is now described. The 
1 5 scheme provides an estimate of the delay and jitter between any arbitrarily specified 
pair of routers based on measurements between a subset of pairs of routers. 

Fig. 4 is a flowchart depicting such a scheme, which includes the three steps 
illustrated therein. 

20 1 . Identification of a minimal subset of router-to-router measurements, Q, that 

must be made in order to obtain an estimate of delay components, x = [dj, dj, 
...,dM, Si,S2, ...,SNf (step 400). 

2. The use of a standard technique to estimate delay components (step 410). 

3. The estimation of the delay between the given pair of routers using an 
25 equation that expresses router-to-router delay in terms of the delay 

components (step 420). 
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The Identification of Router-to-Router Measurements 

Letting Q. be the minimal subset of Q number of pairs of routers between 
which measurements are made, the Q measurement, 7^= Zj^ + v,, = H,, x + v^, k = 1 , 2, 
. . Q, can be re- written as 

y = H X + V (48) 



where 



y = [yi,y2, ••■^yqf 

V= [Vi, V2, Vqf 

H = [Hi H2 . . . Hq 



(49a) 
(49b) 
(49c) 



Thus, a method according to one embodiment of the present invention identifies the 
15 subset Q such that the resulting matrix H has the maximum possible rank, Q = |Q|. In 
such an embodiment, row vectors are Hj, Hj, - . ., Hq are independent. This can be 
represented, for example, in the following manner. 

Fig. 5A is a flow diagram illustrating a method according to one embodiment 
of the present invention that identifies the subset Q such that the resulting matrix H 

20 has the maximum possible rank. The process begins by initializing the set Q. to empty. 
The set Q will contain router pairs between which performance is to be measured 
(step 500). Next, a set 0 is initialized to empty (step 502). This set will contain 
router pairs for which processing time cannot be separately estimated. The first M 
measurements are between all adjacent routers, one for each link, and proceeds as 

25 follows. For each link, Lj,, k = 1, 2, M (step 504), the pair of routers (R,, Rj) that 
are the "end-points" of that link are identified (step 506) and added to the set Q (step 
508). Then, for each router (1, N) , say (step 510), the collection of adjacent 
router are examined to determine where there is a pair of routers adjacent to Rj, such 
that a path between them passes through R^ (step 512). If such a pair is successfully 

30 identified (step 514), then this pair is added to the set Q (step 516). Otherwise, it may 



-42- 

#634677vl 

Client Reference: 53382 Weil #1741 



Attorney Docket No.: M-7915 US 



be concluded that it is not possible to separately assess delay due to processing within 
the router R,,. In this case, is added to the set 0 (step 518). 

Once the sets Q and 0 are thus determined, all the router pairs in CI are 
examined. For each router pair (step 520), row Hj, of the H matrix is determined, such 
5 that = Delay (R;, Rj) = Hj,x (step 522). Once the elements of the H matrix have 
been determined, the H matrix is formed from the elements so determined (step 524). 
Once the H matrix has been formed, the rank Q of the H matrix is determined (step 
526). It can be verified that Q must be N+M - \Ci\. 

The above method for identifying the subset CI such that the resulting matrix H 
1 0 has the maximum possible rank is now outlined using a formal representation. 

BEGIN 

/* proposed set of pairs of routers, between which performance is to be measured */ 
15 Q = 

/* set of routers, ANMT processing time for which cannot be separately estimated */ 

©={}; 

20 /* first M measurements are between adjacent routers */ 
For all links L^, k = 1, 2, . . M, do{ 

(Rj, Rj) = identify_routerjpair(Lk) 
Q = Q + {(Ri,R3)}; 

}; 
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/* N - |0j measurements are between certain pairs of routers adjacent to an Rj */ 
For all routers R^, k = 1, 2, N, do { 
pair_identified = false; 
(p(Rk) = set_of_all_adjacent_routers(Rt); 
5 For all (R^ g (pCRJ) and (R^ e (p(Rk)) and (R^ ^ Rj), do{ 

If (router_path(Ri, R^) = <Ri, R^, Rj>) then { 
Q = Q+ m,R)}; 
pair_identified = true; 

exit /* from the current loop */ 

10 }; 

. )' . 
if pair_identified = false then { 

0 = 0 + {RJ 

} 

15 }; 
k= 1; 

For all pairs of routers (Rj, Rj) e Q do { 

determine such that Delay(Rj, R,) = H,, x 
20 k = k+l 

} 

Form matrix H; 

25 /* Rank(H) should be N+M-|0| */ 
Q = Rank(H) 

END 



30 Before establishing the adequacy of the above method, it is noted that using a 

network performance measurement tool, such as ANMT, allows probe packets to be 
sent through a given router using "loose source routing". In such cases, it is possible 
to ensure that the subset 0 is empty. 

Proof that the (N+M-|0|) x (N+M) matrix, H, resulting from an application of 
35 the above method has full rank (i.e., N+M-|0|) is now given. 

If © is non-empty, the columns of H can be re-arranged by moving columns 
corresponding to the variables Sj for all i, such that R^ e 0, to the end of matrix H. The 
resulting H matrix then has the following structure: 
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H= [ I H,, H,3 ] (50) 
[ H22 H23 ] 

5 where I is an M x M identity matrix, and H12, H13, Hj], H22 and H23 are sub-matrices of 
appropriate dimension (M X N-|e|, M X |0|, N-|0| X M, N-|0| X N-|0|, and N-|0| X 
|0|, respectively). In particular, sub-matrices Hj3 and H23 correspond to variables, S;, 
such that Rj e 0. If 0 is empty, then H13 and H23 are not present. 

The individual rows of the sub-matrix 

10 

[ H,, H22 H23 ] (51) 

are now considered. Each row hj^+k represents a measurement of the type 

1 5 Delay(R,, R,) = s^ + d^ + dj + s, (52) 

where Rp and Rqare routers that are directly connected to router Rj^ using links L; and 
Lj. Consider rows h; and hj, which correspond to measurements: 

20 Delay(Rp, R^) = Sp + d^ + (53) 

Delay(R„R,) = s, + dj + s, (54) 

respectively. Thus, setting row h^+k = (h; + hj - h^+k) yields h^+k =[0 0 ...020 . . .0], 
25 where the "2" is in the M+k position. 

Repeating this operation for every row M+k = M+1 , M+2, . . . , M+N-|0| results 
in a matrix H' of the form: 

H'= [ I H,, H,3 ] (55) 
30 [ 0 2*1 0 ] 

As is thus made evident, the matrix H' has full rank (i.e., Rank(H') = M+N-|0|). 
Because Rank(H') < Rank(H) < M+N-|0|, it follows that Rank(H) = M+N-|©|. 
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Delay between any pair of routers, Delay(Rp, R^), can be expressed as a linear 
combination of the collection of { Delay (R;, Rj), (R,, Rj)e Q}. Equivalently, it can be 
shown that if Delay (R^, R^) = Fj, x, for some F^, then F^ = % H, where the M+N-j©) x 
M+N matrix corresponds to the subset Q.. From Equation 52, matrix H is defined as 



H= [ I H„ ] (56) 

r H,, H,, H,, 1 



Letting 



C= [ I 0 ] (57a) 
[ -I ] 



it may be shown that 

H' = C H = [ I H,3 ] (57b) 

[ 0 2*1 0 ] 

This is so because H21 H12 - H22 = 2*1, and H21 H13 - H23 = 0. Defining: 

B= [ I -I/2H12 ] (58a) 
[ 0 1/21 ] 



H" can be derived as follows: 



H"=BH'=BCH=[ I 0 ] (58b) 

r 0 I 0 1 



As noted, the columns of the sub-matrix, H13, correspond to variables, s^, for each 
30 router, R^, where R^ e 0. Such a router is one through which no route passes, except 
as a source or destination router. 

The delay between any given pair of routers, R^, and Rq, can thus be expressed 
in terms of delay components, assuming that link_path(Rp, Rg) = <L^, L^, . . ., L^> is 
given by: 

35 
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10 



Delay(Rp, R^) = + d„ + + . .. + + (59a) 
= F,x (59b) 

5 where 

Fk=[0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0] (59c) 

and the 1 's in appear in columns u, v, . . ., w, and M+p and M+q. As is now shown 

Fk X = (j)k H" X = (t)k B C H X, (60) 

where (j)^ is simply the first M+N-|0| components of the row vector F^,. To prove that 
Fj, X = H" X = (j)), B C H X, the following cases need to be considered: 

1. Case 1 : If |0| = 0, then, there are no columns corresponding to sub-matrix H13. 
As a result, H" = B C H = I and (t)k = F^ 

2. Case 2: If |0| > 0, then if e ©, the first two terms in the expansion of 
Delay(Rp, Rq) can be combined to yield as Delay(Rp, Rq) = (Sp + dj+ ... + 
d^ + Sq. The u"* row of H" x is exactly (Sp + d„). Similarly, if Rg g @, the last 
two terms in its expansion can be combined to yield Delay(Rp, Rq) = Sp + 
d^, + . . . + (d^ + Sq). The w*** row of H" x is exactly (Sq + d^). As a result, 
entries in the last |0| colunms of F,, may be dropped. 



25 



As a result, from (60) above, F^ x = H x, where 

ai, = B C (61) 



Thus, delay between any pair of routers can be expressed in terms of measured delay 
between pairs of routers, identified by Q. An example employing the preceding 
30 results is now given. 
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The example network given in Fig. 2 is now considered. Note once again that 
N=6, M=9. The routing table is given in Tablel . Using the method described 
previously, the collection of pairs of routers is identified: 



Q = {(R„ R,), (R2, R3), (Ri, R4), (R2, R4), (R2, R5), 
(R3, R5), (R3, R,),(R4, R,), (R5, R,), (R„ R3), 
(R„R,), (R„R3),(R4,R.)} 



(62) 



Further, since no path passes through routers Ri or through Rg, 



0 = {Ri,R6} 



(63) 



Because |0| = 2, the 13x15 matrix H is given by: 



Because 0 = {Rj, R^}, the columns are re-arranged such that the columns 
corresponding to Sj, and are listed last. This results in a new H matrix corresponding 
to a new definition of the vector of delay components, x = [di , dj, . . . , dg, Sj, ...,85,81, 
20 s,]^: 
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then H" = B C H = 



Thus, it can be seen that matrix H" has foil rank (i.e., N+M-|0| = 6+9-2 = 13), and that 
Rank(H) =13. 

The fact that | 0 | = 2, or that the rank of matrix H is 13, and not 15, implies 
that one is not able to estimate all N+M=15 parameters individually. In light of the 
present invention, a close examination of the network reveals that there is no way that 
one can estimate d,, dj, and s,, independently (or d,, dg, and Sg). In every measurement 
that involves dj, parameters Sj and dj appear together (and similarly, dj & Sj, dj & s^, 
dg & Sg). This is so because there is no path in the network that passes through node 
Rj (or through Kg). 

A method according to one embodiment of the present invention, however, 
does provide a measure of dj+Sj, dj+Sj, d7+S6, d^+Sg. This is not a constraint because 
delay between any pair of routers can be expressed in terms of measured delay 
between pairs of routers in Q. In particular. 
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Delay(Ri, Rg) = Si+ds+dg+dg+Sg (64a) 

-F(i,6)X (64b) 

= [0 0100001100001 l]x (64c) 

= [0 0 1 0 0 0 0 1 1 0 0 0 0] H" X (64d) 

5 = 6) B C H X (64e) 

= 0C(i,6)Hx (64f) 

Delay(Ri, R^) = S3+d5+d8+S4 (65a) 

= F(3.4)X (65b) 

10 =[0 0000101001100 0]x (65c) 

= [0 0 0 0 0 1 0 1 0 0 1 1 0] H" x (65d) 

= (t)(3,4) B C H X (65e) 

= a(3 4^ H X (65f) 

1 5 where x = [dj, dj, . . ., d9, Sj, . . . , Sj, Sj, Sg]^, and H, C, and B are given by their 
respective definitions immediately preceding. Thus, 

ct(i,6) = [0 01000011000 0]BC = [0 000000 -1 0001 1] (66a) 

20 a(34) = [00000 1 0 1 00 1 1 0]BC = [00000 1 00-1 000 1] (66b) 



In other words, Delay(Ri, R^) = - Zg + +Z13 and Delay(Ri, R^) = Zg - +Z13, where 
Zk = Hk X corresponds to the k* pair of router in Q. As a result, the delay between any 
given pair of routers can be obtained from the set of orthogonal delay components. 

25 A Formalized Description of a Method for Calculating Delay and Jitter Between an 
Arbitrarily Specified Pair of Routers 

A procedure according to a method of the present invention that computes 
round-trip delay and delay-jitter between any arbitrarily specified collection of router 
pairs using measurements between router pairs in the measurements set is now 

30 described. The theoretical underpinnings of the method used to construct the 
measurements set and the method of calculating delay and jitter from the 
measurements are discussed above, in the description immediately preceding. Such a 
procedure is particularly useful when the number of pairs of routers in the 
requirements set is large. Such a procedure is applicable when measurement errors 

35 are negligible, and a-priori measurements are not available. 
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Fig. 5B is a flow diagram illustrating such a method. It is assumed that the 
network topology and routing table, for a given DS codepoint, is known. Where 
necessary, such a procedure is executed for each DS codepoint under consideration. If 
the network topology and/or the routing table changes, the procedure described 
5 presently should be executed afresh with the new topology or with new routing table. 
The process begins with the specification of the collection of router pairs (step 540) 
and the definition of the delay-components vector (step 542). Next, the sets Q and 0 
are initialized to empty (steps 544 and 546, respectively). The set Q will contain 
router pairs between which performance is to be measured, while the set 0 will 
10 contain router pairs for which processing time cannot be separately estimated. 

The first M measurements are between adjacent routers, one for each link, and 
proceeds as follows. For each link, Lj,, k = 1, 2, M (step 548), the pair of routers 
(Rj, Rj) that are the "end-points" of that link are identified and added to the set Q (step 
550). The next set of up to N measurements are then identified. These measurements 

15 are between a pair of routers (Rj, Rj), each of which is adjacent to router R^. Such a 
pair is included in the measurements set provided there exists at least one path from R; 
to Rj which passes through K^. As a result, this may result in N - |0| measurements 
only. For each router R^, k = (1, N) (step 552), the collection of adjacent routers 
((f)(Rk)) are examined to determine where there is a pair of routers adjacent to R^ such 

20 that a path between them passes through Rj, (step 554). For each router pair in (p(Rk) 
(step 556), the following determination is made. If such a pair is successfully 
identified (step 558), then this pair is added to the set Q (step 560). Otherwise, it may 
be concluded that it is not possible to separately assess delay due to processing within 
the router R,,, and so Rj, is added to the set © (step 562). Construction of the 

25 measurements set, Cl, is thus completed. The size of Q (represented, e.g., by Q) is 
given by N+M-|0| (step 564). 

Once the sets Q and 0 are thus determined, the router pairs in Q are examined. 
For each router pair (step 566), a unique number associated with that router pair and 
row Hk of the H matrix is determined, such that = Delay (R^, Rj) = H^x (step 568). 
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Once the elements of the H matrix have been determined, the H matrix is formed from 
the resulting elements and the rank of H determined (step 570). It can be verified that 
Q must be N+M - |Q|. If such is not the case (step 572), the procedure is exited. 

Next, the colimms of H are re-arranged by moving the columns corresponding 
5 to Sj (for all i, such that g 0) to the end of H (step 574). Also at this time, the 
components of x are re-arranged by moving the components corresponding to s, (for 
all i, such that Rj e ©) to the end of x. Matrices B and C are then constructed, and 
their matrix product D computed (step 576). For each pair of routers n^. = (Rp, R^) in 
the requirements set, O (step 578), construct the row vector, Fj,, and the row vector, cj),. 



10 (step 580). Next, a^^ is computed, ensuring that Delay(Rp, Rq) = F,^ x = H x (step 
582). 



Now that initialization is complete, measurements can now be taken, and delay 
and jitter between router pairs contained in the requirements set calculated. While 
fiirther measxjrement time intervals remain (i.e., measurements are to continue being 

15 taken) (step 584), and for each router pair iiy, = (Rj, R,) in the measurements set, Q 
(step 586), measure average round-trip delay (yjj and delay -jitter (y^ between each 
pair of routers (tTj. = (Rj, Rj)) during the given time interval (step 588). This can be 
accomplished, for example, using an ANMT. Additionally, for each router pair (R^, 
Rq) in the requirements set, O (step 590), calculate round-trip delay vector (A^) and 

20 jitter (aj (step 592). This is repeated imtil no measurement time intervals remain. 

The above method for computing round-trip delay and delay -jitter between 
any arbitrarily specified collection of router pairs using measurements between router 
pairs in the measurements set is now outlined using a formal representation. 



25 



BEGIN 



30 



/* 
/* 
/* 
/* 
/* 



The following assumes that the network topology and routing table, */ 
for a given DS codepoint, is known. */ 
Where necessary, such a procedure is executed for each DS codepoint, */ 
under consideration. */ 
If the network topology and/or the routing table changes, this procedure*/ 
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/* is preferably executed afresh with the new topology or with new */ 
/* routing table. */ 

/* Specify the collection of pairs of routers. */ 

(D = {7r, = (R„R,),k=l,2,...,P}; 
/* Define the delay-components vector. */ 

x = [di,d2, ...,dM, s„S2, ...,s^Y; 

I* Initialize the measurements set (consisting of pairs of routers, */ 
/* between which delay/jitter is measured) to an empty set. */ 

/* Initialize the set of routers (for which ANMT processing delay */ 
/* cannot be separately estimated) to an empty set. */ 

© = {}; 

/* Identify the first M measurements. These are one for each link, */ 

/* and between adjacent routers. It will be noted that the procedure */ 

/* identify_router_pair(L]J identifies the adjacent routers directly */ 

/* linked using link, L,,. */ 

For link (L^) where k = 1 , 2, . . M, do { 
(Rj, Rj) = identify_router_pair(LjJ; 
Q = Q + {(R„R^)} 

}; 



/* Identify the next set of up to N measurements. These measurements */ 

/* are between a pair of routers (R^, Rj), each of which is adjacent to */ 

/* router Rj^. Such a pair is included in the measurements set provided */ 

35 /* there exists at least one path fi-om R; to Rj which passes through R,,. */ 

/* As a result, this may result in N - 10| measurements only. It will be */ 

/* noted that the procedure set_of_all_adjacent_routers(Rk) identifies */ 

/* all routers that are adjacent to the specified router, R^. */ 



40 For all routers R^, k = 1 , 2, . . ., N, do { 
pair_identified = false; 
(p(R,^ = set_of_all_adjacent_routers(Rk) j 
For all R^ e cp(Rk), R^ e (p(Rk), R^ Rj, do { 
If (router_path(Ri, R^) = R^, Rj>) { 
45 Q = Q + {(Ri,Rj)}; 

pair_identified = true; 
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exit 

}; 

If (pair_identified = false) { 
© = © + {RJ 
5 } 

}; 

/* Construction of the measvirenaents set, Q, is now complete. */ 

/* The size of the set is N+M-|0|. */ 

10 

/* For each pair of routers in the measurements set, (R^, Rj), */ 
/* associate a unique number, and determine the corresponding row vector,*/ 

15 /* Hk such that round-trip delay between the router pair (R,, Rj) */ 

/* can be expressed as x. */ 

k=l; 

For all pairs of routers (Rj, R,) e Q, { 
20 determine such that Delay(Ri, Rj) = x; 

k = k+l 

}; 

/* Form the matrix, H, and compute its rank. */ 
25 /* The Rank(H) should be Q = N+M-|0|. */ 

H-[HiTH/...HqY; 

If {Rank(H) != Q) { 

30 exit /* EXIT remaining procedure */ 

} 

/* Re-arrange columns of H as follows: move colimms corresponding to */ 
/* variables, s^, for all i, such that Rj e 0, to the end of matrix H. */ 
35 /* Similarly, re-arrange the components of the vector of */ 
/* delay components, x. */ 

H= [ I H,, H,3 ]; 
[ H,, H,, H,3 ] 

40 

/* Construct matrices, B and C, and compute the matrix product D = BC. */ 

C= [ I 0 ]; 

[ H,, -I ] 

45 

B = [ I - 1/2 ]; 
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[ 0 1/21 ] 
D-B C; 

5 /* For each pair of routers = (Rp, Rq) in the requirements set, O, */ 

/* construct the row vector, and the row vector, */ 

Construct F^ such that Delay(Rp, R^) = F^ x; 

10 From F^ construct (j)^ by retaining the first Q components of F^; 

/* Compute a^. This ensures that Delay(Rp, Rg) = F^ x = aj, H x. */ 

15 

/* The initialization steps are now complete. */ 
/* Measurements repeatedly and calculate delay and jitter between router */ 
/* pairs contained in the requirements set. */ 

20 /* tg is the start time and dt is the measurement interval. */ 
to = start_time; 

while (! finished) { 

For eachk= 1,2, ...,Q,do { 
25 During the time interval, (to, to +dt), use ANMT to measure average 

round-trip delay, y^, and delay-jitter, y^,, between each pair of routers, 
% = (Rj, Rj) contained in Q 

}; 

30 /* For each pair of routers, n^. = (Rp, R^), in the requirements set, <I), */ 
/* calculate round-trip delay vector, Aj^, and jitter, cj^. */ 

For each k= 1,2, \0\,do { 
\ = 2i=i,...,Q (ak,i yd; 
35 a, = VE,.i_...,Q(a,/yi2) 

}; 

/* Repeat the above for the next time interval. */ 
to = to+dt 

40 

} 

END 
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III. Estimating Delay and Jitter in an Enterprise Network Coupled to a 
Backbone Having an Unknown Topology 

The various embodiments of a method according to the present invention 
described previously provide an efficient measurement scheme to estimate round-trip 
5 delay and jitter in an enterprise network whose topology and routing is known. These 
methods may be extended to monitor delay and jitter in an enterprise network that 
uses a service-provider (SP) backbone network having topology and routing that are 
not known. An embodiment of the present invention, such as is now described, 
illustrates how the preceding method can be used in networks where enterprise routers 

10 are connected through a backbone network of a service provider (SP), the topology or 
routing of which is vinknown (and cannot be discovered). In this portion of the 
discussion, the network is referred to as an enterprise network in order to distinguish 
the enterprise network from the service provider's backbone. In a similar fashion, 
elements of the enterprise network are referred to as being a part thereof (e.g., 

15 enterprise routers and enterprise links). 

More specifically, a method according to one embodiment of the present 
invention discovers the collection of ingress/egress routers of the backbone or service- 
provider network used to intercormect the enterprise router. Based on the resulting 
incomplete knowledge of backbone network topology, a model of the backbone 
20 network together with the manner in which enterprise traffic is routed through the 
service-provider network can be generated. Such a measurement scheme: 



1. 



can be used in the current context to estimate delay and delay -jitter between 
specified pairs of enterprise routers. 



25 



2. 



can be used to estimate various delay components between a given pair of 
enterprise routers, and 



3. 



can provide an efficient method estimating delay and jitter between a large 
number of pairs of enterprise routers. 
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Thus, a relatively simple method is described that identifies the relevant set of 
backbone routers and their interconnection to enterprise routers. Such a method 
allows the modeling of the relevant portion of the backbone network together with its 
interconnection to enterprise routers. The resulting model of the network is 
5 characterized by N+m routers and N+m*(m-l)/2 links, where m is the number of 
backbone routers and N is the number of enterprise routers. It can be shown that, if 
the requirements set contains P pairs of enterprise routers, then the size of a 
measurements set is no more than min(P, N+m*(m-l)/2). Because m is the number of 
backbone routers, the technique is particular efficient if the extent of "clustering" is 
10 significant, or equivalently, m is small. 

A Model of the Enterprise Network 

The enterprise network is assumed to include N routers, R^, i = 1, 2, . .., N, that 
are connected to one or more backbone routers using links, Lj, j = 1 , 2, . . ., N. It is 
assumed that the number of links used to connect an enterprise network to the 
15 backbone network is 1, and that the enterprise routers are themselves not connected to 
each other. While these assumptions are not mandatory (i.e., such topological features 
can be accounted for), such are assumed for simplicity of the following discussion. It 
is also assumed that (a) the identity of backbone routers, and (b) the interconnection 
of enterprise routers to the backbone routers are unknown. 

20 Fig. 6 is a block diagram illustrating an exemplary enterprise network 600 in 

which a method according to an embodiment of the present invention can be 
practiced. Network 600 includes a backbone 610 and routers 620(1 )-(K) and 620(K)- 
(N). Routers 620(1)-(K) and 620(K)-(N) are coupled to backbone 610 by links 
630(1)-(K) and 630(K)-(N), respectively. It will be noted that routers 620(1)-(K) and 

25 620(K)-(N) are also designated R;, i = 1, 2, ...,K, N, and that links 630(1)-(K) and 
63 0(K)-(N) are also designated Lj, j = 1 , 2, . . . ,K, . . . , N. This is done to allow the 
formulation of the equations used to describe various embodiments of methods 
according to the present invention. 
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The first step towards identifying the subset of pairs of routers for which 
measurements should be made involves discovering the identity of backbone routers 
and their interconnection to enterprise routers. In doing so, a method according to one 
embodiment of the present invention, such as the one given below, also partitions the 
5 collection of enterprise routers, R;, i = 1,2, . . . , N, into one or more enterprise 
"clusters," C^, k = 1,2, . . . , m, such that each enterprise router, R^, i = 1 , 2, . . . , N, 
belongs to one and only one cluster. By definition, each cluster, C^, k = 1,2, . .., m, 
contains % number of enterprise routers, re-labeled as R^kj)) j = 1, 2, . . ., n,,, such that 
each router R^^ jj in the cluster, C^, is connected to the backbone router, p,^, using a 
1 0 link L(j^ J), j = 1,2, . . . , Uj^, k = 1,2, . . . , m. Such clusters may be identified using a 
process such as that described below. 

Fig. 7 is a flow diagram illustrating the operations performed in the process of 
forming "clusters" (aggregations of network elements), as described above. It is noted 
that each cluster is identified by the domain-name (DN) of the corresponding 

15 backbone router, for simplification, although this need not be the case. Initialization 
of the set that will contain the collection of clusters is performed (step 700). For each 
router (step 705), then, a different router is selected (step 710) and the sequence of 
network addresses along a path to the selected router is determined (step 715). Once 
the sequence of addresses has been determined, a reverse domain-name look-up is 

20 performed on a name server for the first backbone router, as identified by the network 
address of the router/port (step 720). If the domain-name of the first backbone router 
is among the domain-names corresponding to the collection of clusters already 
identified, then the current router is simply added to the given cluster (step 730). If 
the domain-name of the first backbone router is not in the set of clusters (step 725), 

25 the domain name of the new backbone router so discovered is added to the set of 

backbone routers (step 735). Additionally, a new cluster is created within the set of 
clusters (step 740) and the current enterprise router added to the new cluster (step 
745). 

Once all routers have been examined thusly (for each domain-name in the set 
30 of clusters or domain-names (step 750)), several actions are then taken. These actions 
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include the storage of a number representing the number of elements in the given 
cluster (as denoted by the given domain-name) (step 755); the re-labeling of the 
backbone router in the present cluster (step 760); and the re-labeling of the enterprise 
routers in the present cluster (step 765). Once all of the domain-names in the set of 
5 domain-names have been processed, the number of clusters is determined, and then 
stored (step 770). 

Given a collection of enterprise routers, 9? = {R^, i = 1,2, . . ., N}, such as that 
presented as network 600 in Fig. 6, the grouping of routers (e.g., routers 620(1)-(K) 
and 620(K)-(N)) can be described in a more formal manner. The process referred to 
10 herein as form clusters is one such description. 

BEGIN "form_clusters" 

15 /* S is a collection of clusters. */ 
/* Each cluster is ID'd by the domain-name of corresponding backbone routers. */ 



2 = {}; 



For i= 1,2, ...,Ndo { 



20 



Select a router, R,, i j; 



/* Rj may be selected arbitrarily. 



25 



/* Determine the sequence of IP addresses (or domain names) 
/* of routers/ports along the path to router R^. 



*/ 
*/ 



<Sj, Sj, Sk> = traceroute(Ri, Rj); 



/* 11 = DN of first backbone router, ID'd by IP addr or DN of router/port, Sj */ 



30 



IX = reverse_NS_lookup(Si); 
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if H € S { 
then 

/* is uniquely ID'd by domain-name ja of backbone router */ 
add_to_cluster(Ri, C^); 

5 else { 

add_to_SP_routers(|j,, E); /* New backbone router found */ 
C,= {}; 

add_to_cluster(R„ C^); 

} 

10 } 
k = 0 

For all n e S do { 
k=k+l; 

n^ = I I; /* I C I denotes the number of elements in set C */ 

1 5 Re-label the backbone router |li as p^; 

Re-label routers in cluster as R^^. q, i = 1,2, .. ., n,^; 
} 

m = k; /* m is the number of clusters \ */ 

20 END "form_clusters" 



As a result of forming clusters (and re-labeling of enterprise routers), a 
acceptably complete description of the interconnection between enterprise routers and 
SP-provided backbone routers is generated. The resulting topology, corresponding to 
25 the enterprise network of Fig. 6, is illustrated in Figs. 8A and 8B. 

Fig. 8A is a block diagram illustrating the topology of a synthesized network 
800, depicting an inferred backbone topology generated by a method according to an 
embodiment of the present invention. Network 800 includes a backbone 810, the 
topology and routing of which are unknown to the methods described herein, aside 

30 from the ability to detect ingress router. In Fig. 8A, routers 620(1)-(K) and 620(K)- 
(N) are grouped into clusters 820(l)-(m) and appear as routers 830(1, l)-(m,nm). 
Routers 830(l,l)-(m,nm) are coupled to backbone 810 by links 840(1, l)-(m,nm), 
respectively. In fact, as depicted in Fig. 8 A, links 840(1, l)-(m,nm) couple routers 
830(l,l)-(m,nm) to a number of backbone routers 850(l)-(m). Backbone routers 

35 provide points of ingress or egress to backbone 810 for the enterprise network 
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connected thereto. As noted, the only infonnation that can be gleaned from backbone 
810 is the existence of backbone routers 850(l)-(m) — further information as to the 
topology and routing of backbone 810 remains unknown to the methods described 
herein. It will be noted that routers 830(1, l)-(m,nm) are also re-designated R^y) (i = 1, 
5 2, ...,k, m; j = 1, 2, ...,ni), and that links 840(1, l)-(m,nm) are also designated L(ij) ( 
i = 1,2, ...,k, ...,m;j = 1,2, ...,ni). Additionally, it will be noted that clusters 820(1)- 
(m) are also designated C;, i = 1, 2, ...,k, m, and that backbone routers 850(l)-(m) 
are also designated p^, i = 1, 2, . ..,k, m. This is done to allow the description of the 
parameters in terms of the equations and methods described herein. 

10 Knowledge of topology and routing in the enterprise network may still be 

incomplete, however, because details concerning topology and routing within the SP 
backbone network are unavailable. This might be due, for example, to the fact that 
such a backbone network may not forward probe packets (e.g., traceroute packets) 
beyond the ingress router. As a consequence, discovery of the routes used within the 

1 5 backbone network may not be feasible. The backbone network may therefore be 

modeled as a fully-coimected backbone network that includes backbone routers and 
backbone links. 

Fig. 8B is a block diagram illustrating network 800, with routers 830(1,1)- 
(m,mn) coupled to backbone 8 1 0 by links 840(1 , l)-(m,nm), respectively, as in Fig. 

20 8A. As before, routers 830(l,l)-(m,nm) are coupled to backbone routers 850(l)-(m), 
which provide points of ingress or egress to backbone 810 for the enterprise network 
cormected thereto. As noted, the only definitive information that can be gleaned from 
backbone 810 is the existence of backbone routers 850(1 )-(m) ~ further information 
as to the topology and routing of backbone 810 remains unknown to the methods 

25 described herein. However, certain assumptions can be made that allow the topology 
of backbone 810 and its routers to be modeled. This allows delays due to actual 
backbone links to be identified, or at least portions of delays to be attributed to either 
backbone links or enterprise links. 
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Backbone 810 is depicted as including a number of backbone routers (i.e., 
backbone routers 850(1 )-(ni), as before). However, in Fig. 8B, each of the backbone 
routers is connected to each of the other backbone routers by a backbone link 
(backbone links 860(1, 2)-(m-l,m)). As noted, backbone links 860(1, 2)-(m-l,m) are 
5 virtual, but are useful in allocating delay throughout network 800. Thus, backbone 
810 is represented as having a Mly-connected topology generated by a method 
according to an embodiment of the present invention. As before, backbone routers 
850(l)-(m) are also designated p„ i = 1, 2, ...,k, m. Moreover, backbone links 
860(1, 2)-(m-l,m) are also designated X(i,j) (i = 1, 2, .. .,k, (m-1); j = (i+1), m), i 
10 ^']. This is done to allow the description of the parameters in terms of the equations 
and methods described herein. The backbone network may therefore be modeled as a 
fiilly-connected network that includes routers, p^, k = 1, 2, m, and links, A,(],i,k2)' 

kl, 

k2= 1,2, ...,m,kl ^ k2. 



light of the alternative embodiments now described that estimate the round-trip delay 
and jitter for a specified set of pairs of routers in the case where the network includes 
a service-provider network of unknown topology and routing. 

First, the user-specified requirements set, O, is defined as a collection of P 
20 pairs of routers for which delay and delay-jitter metrics are needed, as before. Also as 
before, the subset is normally a small fraction of all possible pairs of routers. The 
user-specified requirements set can be defined as: 



Definitions 



15 



Several of the variables and notations discussed previously are now defined in 



<D= {7rK,K= 1,2, ...,P} 



(67) 



25 



where 



T^K - (R(kl,i)' ^j)) 



(68) 
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Because round-trip delay or jitter is monitored, it is immaterial whether the pair of 
routers is specified as tik = (R<ki,i), R(k2j)) or as tTk = (R(k2j). R(ki,0')- Obviously, if only 
one direction were measured and asymmetries existed, this would not be the case. 

Second, the measurements set, Q, is a collection of Q number of pairs of 
5 routers for which roimd-trip delay and delay-jitter are actually measured using an 
ANMT, for example. A fundamental property of the measurements set is that, based 
on measurements so obtained, one can accurately estimate delay/jitter metrics for all 
pairs of routers specified in the requirements set, O. In particular, the measurements 
set can be defined as 

10 

Q={7r'K,K=l,2, ...,Q} (69) 

where 

^ 15 7i'k = (R(,u),R(^j)) (70) 

As will be apparent to one of skill in the art, Q will always be a subset of O. 

It is assumed that, for a given pair of enterprise routers, R(ki,L) and R^mj)^ the 
only available route between them is through the backbone routers, p^i and p^j, that 
20 physically connect R^j and R(k2j) to the SP backbone, respectively. As a result, the 
route between them is given by 

Link_path(R(ki,i), R^j)) = <L(ki.i), ki), L(k2j)> (71 ) 

25 

Three types of delay components that contribute to the round-trip delay between a pair 
of enterprise routers may be defined: 

1 . d^ i), k = 1 , 2, . . . , m, i = 1 , 2, . . . , n^, the round-trip delay in transferring an IP 
packet between the pair of routers, R^y -,^ and p^, over the link, L^^, 
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2. i), k = 1 , 2, . . . , m, i = 1 , 2, . . . , Ilk, the ^^^^ it takes to generate, receive and 
process a probe packet within transport and apphcation layers in an enterprise 
router, and 

3 • C(ki k2)' kl , k2 = 1 , 2, . . . , m, kl 9^ k2, the round-trip delay in transferring an IP 
5 packet between the pair of backbone routers, p^i and pi^. 

This definition builds on that previously given with regard to protocol transfer delay 
and processing delay. 

The ordered collection of delay components is re-written in the form of a 
1 0 vector of size 2N+m* (m- 1 )/2 : 



20 



[d(U) 


^^(1,2) ••■ 


d(l.nl) 


d(2,i) 


d(2.2) ■•■ 


'^(2,n2) 




dim,2) •■■ 




C(l,2) 


^^(1,3) ... 


m) 


C(2, 3) 


C(2,4) 


C(2, m) 




> 




^(1,1) 


^(1,2) 


^(l,nl) 


S(2, 1) 


S(2,2) 


^(2, n2) 




S(m, 2) 





25 In view of the definition of delay components set out above, the round-trip 

delay between a given pair of enterprise routers, R^ki,i) and R^j), as seen by network 
monitoring applications, is given by: 
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(a) in the case where kl ^ k2, and assuming that Hnk_path(R^, R(k2j)) = '^^(yi;xY 

ZK = Delay(R(,i,o,R(k2j)) (73a) 

5 = S(ki4) + d(kj^,) + C(ki, + d(k2j) + S(k2j) (73b) 

= FkX (73c) 

where 

10 Fk=[0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0] (73d) 

is a row vector of size 2N+m*(m-l)/2, and the 1 's in Fk appear in columns 
corresponding to column variables, z^x^viy '^(^u)^ d^j^j)' ^^d S(k2j); 

(b) in the case where kl = k2, and assuming that path(R(ti j), R^j)) = ^L^^i,;), L(kij)>, 

15 

z,= Delay(R(ki,i),R<kij)) (74a) 

= S(ki,) + d(ki,i) + d(kij) + S(kij) (74b) 

= FkX (74c) 

20 where 

Fk=[0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0] (74d) 

and the 1 's in F^ appear in columns corresponding to colimm variables, dj^i i), s^^i i), 
25 d(kij), and S(kij). 

Because delay, Zk= Delay(R^i q, R^j)), varies, one may define delay-jitter 
between a pair of routers, }) and R(k2j) as follows: 

Delay-Jitter(R(ki,o, W = < E{(Zk - E{zk})^ } (75) 

30 

where Zk = Delay (R^j^i i), R(k2j)) and E{.} is the expectation operation. The round-trip 
delay between the specified pairs of routers in <1) = {tTk = (R(ici,05 R<k2j))5 k = 1, 2, .. ., 
P}, may be re- written as a vector equation: 

35 z = F X (76) 
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(77) 



[Z2 ] 
[... ] 
[Zp ] 



[F, ] 

[F, ] 

[••• ] 

[Fp ] 



(78) 



The P X 2N+m*(m-l)/2 matrix, F, helps determine the subset of pairs of 
routers between wliich delay measurements are necessary. If Fj, Fj, - . ., Fq is indeed a 
15 maximal set of linearly independent rows of F, then row vectors, Fq+j, Fq+j, • • ., Fp can 
be expressed as a linear combination of Fj, Fj, . - Fq as follows: 



Fk = ..,Q (ttKi Fi), K = Q+1, Q+2, 



(79) 



The constants ; may be re-organized in the form of a row vector (of size Q) 
as follows: 



aK= [ctKi, aK2, ockq], k = Q+1, Q+2, 
25 while the P X Q matrix, A, is defined as: 



[I 

[OCq+2 
[••• 

[ctp 



where the matrix, I, is a QxQ identity matrix. 
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A Measurement and Estimation Scheme for a Specified Requirements Set 

A procedure according to one embodiment of the present invention is now 
described that computes round-trip delay and jitter for every pair of routers, tz^ = 
(R(ki,i)j R<k2j)) contained in a specified requirements set, ^, using measurements 
between pairs of routers in the measurements set, CI, and identified using a method 
such as that described below. The scheme is particularly useful when the number of 
pairs of routers in the requirements set is not large. 

Fig. 9 is a flow diagram illustrating a process that computes round-trip delay 
and jitter for every pair of routers contained in a specified requirements set using 
measurements between router pairs in the measurements set. The computation of 
round-trip delay and jitter (also referred to herein as delay-jitter) begins with the 
specification of the requirements set (step 900). Next, the delay-components vector 
(x) is constructed per Equation (72) and associated equations (step 905). For each 
router pair in the specified requirement set (step 910), the following actions are 
performed. First, the given router pair is identified as being an element of the 
collection of router pairs (step 912). Next, path information is obtained for the given 
pair of routers (step 914). Finally, the row vector (F^ is then determined (step 916). 

Once these steps have been carried out for each router pair in the specified 
collection of router pairs, the F matrix is constructed in accordance with Equation (78) 
and associated equations (step 920). Once the F matrix has been constructed, a 
maximal set of independent rows of the F matrix are identified (step 925). Once 
identified, the number of independent rows in the F matrix is stored (Step 930). The 
rows of the F matrix (as well as the router pairs in O) are reordered such that the first 
Q rows of the F matrix are independent (step 935). For each dependent row of the F 
matrix (step 940), the constants used to express the dependent row in terms of the 
independent rows are determined (step 942), and a corresponding row vector is 
constructed for these constants (step 944). Then the fnst Q pairs of routers 
(representing the first Q independent rows of the F matrix) are copied in order to form 
the subset Q, representing the measurements set (step 945). As has been noted, the 




Client Reference: 53382 Weil #1741 



Attorney Docket No.: M-7915US 



quantity Q represents the number of rows in the maximal set of independent rows of 
the F matrix. 

At this point, the values for the start time and measurement interval are set 
(step 950). Next, while the measurement process remains unfinished (step 955), the 
following actions are performed. For each independent row of F (step 958), the 
average round-trip delay and jitter between pairs of routers in Q are measured during 
the time interval (step 960). Next, for each router pair in O (step 965), the round-trip 
delay vector is estimated (step 970). In a similar fashion, for each router pair in <D 
(step 975), the round-trip delay-jitter is computed (step 980). This completes the 
exemplary process of computing round-trip delay and jitter for the routers contained 
in the specified requirement set. 

In using the above method, the measurement error is assumed to be negligible. 
Moreover, delay estimates are optimal provided no other delay measurements or a 
priori estimates are available. The process of computing roimd-trip delay and jitter 
for every pair of routers contained in a specified requirements set using measurements 
between router pairs in the measurements set is now outlined using a formal 
representation. 



BEGIN 

/* Specify collection of router pairs in requirement set. 

/* Such a pair of router should not be, for example, (RQ^^^y Pi^) 

O = {TtK = (IVi.i), R(k2j)), K = 1, 2, . . ., P}; 



Construct delay-components vector, x; 



*/ 



/* see Eqn (72) and associated equations */ 



For each k = 1, 2, P, do { 

Identify the pair of routers tTk = (R^ki i), R(k2j)) ^ 

Obtain link_j)ath(R(,i R(k2j)) = <L(ki,D, ^(kU2), hnj)>; 

Determine the row vector, F^, such that Zk= Delay (R^i i), R(k2j)) = Fk x 
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Construct P x 2N+m*(m-l)/2 matrix, F; /* see Eqn (78) and associated equations */ 

Identify a maximal set of independent rows of F; 

5 Q = the number of maximal set of independent rows of F; 

Reorder rows of F and router pairs in O such that the first Q rows, Fj, Fj, . . ., Fq, are 
independent; 

10 For each K = Q+1, Q+2, ...,P,do{ 



For each k = 1 , 2, . . ., Q, do { 

During the time interval (t^ to +dt) use a network monitoring tool to 

measure average roimd-trip delay, yK, and delay-jitter, y^, between the 

pair of routers, = (R^,,^, R^.y) 
For each pair of routers in O, estimate round-trip delay vector, 

For each k = 1, 2, . . ., Q, do { 



Determine a^^, a^j. • • oIkq such that F^ = E,=i ^ 
Construct the row vector = [Uk^i, 0.^ 2, . . . , q] 



(aK,F,); 



}; 



1 5 Copy the first Q pairs of routers from O to form the subset 
Identify the start time to, and the measurement interval, dt; 



While (! finished) { 



25 




For each k = Q+1, Q+2, . . ., P, do { 
= . Q (ttKi yi) 

}; 



30 



For each pair of routers in <I>, compute round-trip delay-jitter, a^, 



For each k = 1, 2, . . Q, do { 

}; 



For each k = Q+1, Q+2, . . ., P, do { 



35 



aK-Vi:,.,..,Q(a./Y.2) 



}; 
} 



END 
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The above method may be improved by recognizing that the variables, Sf^, 
and d^i always appear together as s^^,,,) + d^^i^^. In that case, the variable x may be 
redefined as follows (see equation also (72) and associated equations): 

d(2, 1)+S(2, 1) d(2,2)+S(2^2) ■•• d(2,n2)+S(2,n2) 

d(n,,l)+S(„,,l) d(^,2)+S(m,2) •■- d(^, nm)+ V. mn) 

C(l,2) C(i_3) C(i 
C(2,3) C(2,4} ... C(2,m) 



As a result, one may redefine = d^; j^+s^; for all i and j, and represent delay, 
Delay(R(ki R(k2j)), in terms of vector, x, as follows: 

(a) in the case where kl ^ k2, 

Zk = Delay(R(k,,i), R^^sj)) 

= FkX 

where 

Fk=[0 ... 0 1 0 ... 0 1 0 ... 0 1 0 ... 0] (83d) 

is a row vector of size N+m*(m-l)/2, and the 1 's in Fk appear in columns 
corresponding to column variables, D(ki D^^^ and c^^.^^^^ 



(83a) 
(83b) 
(83c) 
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(b) 



in the case where kl k2. 



ZK=Delay(R(ki,i),R(kij)) 

= FkX 



(84a) 
(84b) 
(84c) 



where 



Fk= [0 ... 0 1 0 ... 0 1 0 ... 0] 



(84d) 



is a row vector of size N+m*(m-l)/2, and the ones in Fk appear in columns 
corresponding to variables Dg^j ^ and D^^jj^. 

An Exemplary Network 

Fig. 1 OA is a block diagram illustrating an exemplary network 1000, where N 
= 7. Network 1000 includes a backbone 1010 and routers 1020(l)-(7). Routers 
1020(l)-(7) are coupled to backbone 1010 by links 1030(l)-(7), respectively. It will 
be noted that routers 1 020(1 )-(7) are also designated Rj, i = 1, 2, ...,7, and that links 
1030(l)-(7) are also designated Lj, j = 1, 2, .. .,7, in order to allow network 1000 to be 
discussed in terms of the parameters discussed herein. The only information that can 
be gleaned from backbone 1010 is the existence of backbone routers that allow ingress 
of network traffic into backbone 1010 ~ further information as to the topology and 
routing of backbone 1010 remains unknown. It is desirable to determine, or at least 
estimate, the delay experienced by network traffic flowing through the enterprise 
network, including that transiting backbone 1010. This information may be derived 
using the techniques just described, and results in the scenario depicted in Fig. lOB. 

Fig. lOB illustrates exemplary network 1000 after being apportioned into 
clusters using a method according to one embodiment of the present invention. As 
before, network 1000 includes routers 1020(l)-(7) coupled to backbone 1010 by links 
1030(l)-(7), respectively. It is xmderstood that backbone routers provide points of 
ingress or egress to backbone 1010 for the enterprise network connected thereto. As 
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noted previously, the existence of backbone routers 1050(l)-(4) can be deduced from 
the interaction of backbone routers 1 050(1 )-(4) with routers 1 020(1 )-(7), but further 
information as to the topology and routing of backbone 1010 remains unknown, as 
noted. 

In order to allow the determination of delay and jitter throughout network 
1000, embodiments according to the present invention apportion the enterprise 
network into clusters using a method of cluster formation such as that described 
previously. Using such a method, the enterprise network can be modeled as having m 
= 4 clusters. These are depicted in Fig. lOB as clusters 1040(l)-(4), and are also 
designated as clusters Cj, C2, C3, C4, in order to facilitate the discussions herein. 
Backbone 1010 is depicted as including a number of backbone routers (i.e., backbone 
routers 1 050(1 )-(4)), with the routers in each of clusters 1 040(1 )-(4) coupled to one of 
the a corresponding backbone routers 1050(l)-(4). Moreover, each of backbone 
routers 1050(l)-(4) is coimected to each of the other ones of backbone routers 
1050(l)-(4) by a given backbone link (backbone links 1060(1, 2)-(3,4)). As noted, 
backbone links 1060(l,2)-(3,4) are virtual, but are useful in allocating delay and jitter 
throughout network 1000. Thus, backbone 1010 is represented as having a fully- 
connected topology generated by a method according to an embodiment of the present 
invention. As before, backbone routers 1050(l)-(7) are also designated P;, i = 1, 2, . . ., 
4. Moreover, backbone links 1060(l,2)-(3,4) are also designated X(ij) (i = 1, 3; j = 
i+1, 4). This is done to allow the description of the parameters in terms of the 
equations and methods described herein. The backbone network may therefore be 
modeled as a fully-connected network that includes routers, Pk, k = 1 , 2, . . ., 4, and 
links, >t(ki,k2), kl, k2 = 1, 2, 4, kl ^ k2. It will also be noted that R;, i = 1, 2, 7, 
have been renumbered to reflect their designations in the following calculations and 
appear as R^j j), R^j 2), R(i 3), R(2,i), R^j^i), R(4^i) and R(4 2), respectively. 
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As can be seen, there is a unique route between any given pair of enterprise 
routers. In particular, the route between R(ki^i) and R^j^j^ is given by: 



Link_path(R^i^,), R^^)) = <L(ki,i), 



'(kl,k2)5 Mk2jr 



(85) 



15 



The execution of the method to estimate round-trip delay and jitter for the specified 
requirements set, <D, is now illustrated. Let 



(R(2,l)? 1^1,1))' (R(2,I)? ^l.2)% (^2,ip R(l,3))' 

(R<3,1)5 ^l,l)X (^3,1)J ^1,2))5 (R<3,1)^ I^l,3))j 

(R(4,l)5 R(l,l)), (R(4,l)> R(l,2))= (R<4,1> R(l,3))5 

(R<4,2)5 (R(4,2)5 ^^1,2))5 (R(4,2)? ^1,3)) 
} 



The delay-component vector. 



(86) 



[D(i,i) 

[D(l,3) 

[%,) 

[D(4,2) 
['^(I. 2) 

3) 
4) 

[C(2, 3) 
[^^(2,4) 
[^^(3,4) 



(87) 



Because z, = De\ay(R^^ ,^, R^, J = D^^ + C(,_ + D^,^ 
F, = [1 00100010000 0] 
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Similarly, one may obtain F;, i = 2, 3, . . 12. The resulting 12x13 matrix F is given 
by: 





D(U) 




1^(13) 


D(2 1) 


D(3,l) 


D(4.1) 


D(4,2) 


C(l 2) 


C(l 3) 


C(14) 


C(2 3) 


C(2 4) 


%.4) 




1 






1 








1 












Zj 




1 




1 








1 












Zj 






1 


1 








1 












z, 


1 








1 








1 










Z5 




1 






1 








1 
















1 




1 








1 










Z7 


1 










1 
















Zg 




1 








1 
















Z9 






1 






1 
















Z,o 


1 












1 














Z„ 




1 










1 














Zl2 






1 








1 















It can be verified that Fj, F2, F3, F4, F7, Fjo are independent row vectors. As a result, Q 
= 6. The row vectors, Fj, are re-ordered, as those corresponding to Zj, Z2, Z3, Z4, Zy, z^q. 
The re-ordered version of F is: 









D(t,3) 


D(2,l) 






D(4,2) 


C(l,2) 


'=(1.3) 




C(2,3) 


C(2,4) 


C(3,4) 


Zi 


1 






1 








1 












Z2 




1 




1 








1 












Z3 






1 


1 








1 












Z4 


1 








1 








1 










Z7 


1 










1 








1 








Z,o 


1 












1 






1 








Z5 




1 






1 








1 










Zg 






1 




1 








1 










Zs 




1 








1 








1 








Z9 






1 






1 








1 








Zn 




1 










1 






1 








Z12 






1 








1 






1 
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Further, it can be shown that 



F5 = F2 + F4-F, (89a) 

F, = F3 + F4-F, (89b) 

F, = F2 + F,-F, (89c) 

F, = F3 + F, - Fi (89d) 

F„ = F2 + F,o-F, (89e) 

Fi2 = F3 + F,o-Fj (89f) 



Consequently, 



= [-1 1 0 1 0 0] (90a) 

a, = [-101 100] (90b) 

ag = [-1 1 0 0 1 0] (90c) 

a9 = [-101010] (90d) 

a,i = [-l 1 0 00 1] (90e) 

a,2=[-10 100 1] (90f) 



As a result 



Q= { (91) 

(R(2,l)5 (^2,1). f^l,2)). 

(R(2,l)5 R(l,3))5 (R(3,l)= 1^1,1))' 

(R(4,l). (R(4,2> R(l,l)) 
} 



Initially 

to = 0 (seconds) (92) 

and 

dt = 60 (seconds) (93) 



which can then be used to define an interval (t^ to+dt). During the interval (to, to+dt), 
a network monitoring tool is used to measure round-trip delay and jitter between pairs 
of routers identified by Q, to yield delay measurements, yi, yj, y^, y^, y^, yjo, and 
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delay-jitter measurements, y,, 72, y^, y^, y,, y^^. An estimate of delay between pairs of 
routers in O is given by 



Ai = yi (94a) 

A2 = 72 (94b) 

A3 = Yj (94c) 

A4 = 74 (94d) 

A7 = yy (94e) 

Aio = Yio (94f) 



A5 = y2 + 74 - Yi (95a) 

Ae = 73 + 74 - 71 (95b) 

As = 72 + 77 - 7i (95c) 

A9 = 73 + 77 - 7i (95d) 

A„=72 + 7io-7i (95e) 

A12 = 73 + 710 - 71 (95f) 

An estimate of delay -jitter between pairs of routers in O is given by 

= Ti (96a) 

<^2 = I2 (96b) 

(^3 = 73 (96c) 

= 1a (96d) 

<^7 = 77 (96e) 

cfio = 7io (96f) 



^5 = V(72' + y4' + 7i') (97a) 

<^6 = V(73' + y4' + 7i') (97b) 

^s = V(y2' + 77' + 7i') (97c) 

f^9 = V(y3' + 77' + 7i') (97d) 

f^ii = V (ys' + yio' + 7,') (97e) 

^i2=V(73' + 7io' + 7i') (97f) 

Once complete, the steps of delay measurement and estimation can be repeated by 
incrementing t^ (e.g., tg = to+dt). 
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IV. Estimating Delay and Jitter in an Enterprise Network Coupled to a 
Backbone Having an Unknown Topology and Estimating Delay 
Components Within the Backbone 

A measurement scheme according to one embodiment of the present invention 
is now described that allows one to not only estimate the delay between a given pair 
of enterprise routers but, more importantly, to apportion the same to various links 
along the route, including the SP-provided backbone. This allows the determination 
of whether congestion (if any) exists within the backbone or over the local (or remote) 
links. It is assumed that a model of the enterprise network is available. That is, an 
enterprise network exists and consists of enterprise routers that are interconnected 
through a backbone network provided by a service provider (as in Fig. 8A). It is 
further assumed that a model of the backbone network in terms of backbone routers 
and virtual links (as in Fig. 8B) has been obtained using methods previously 
described. As such, such a network is modeled as a collection of m clusters of 
enterprise routers, with a backbone router corresponding to each. 

Fig. 1 1 is a flow diagram illustrating the operations performed in the process 
of apportioning delay components. The apportioning of delay components begins 
with the specification of a router pair between which delay and jitter are to be 
monitored (step 1 100). Once the routers of the router pair are specified, the clusters 
corresponding to these routers are then identified (step 1 105) and the size of those 
clusters is determined (step 1110). The size of the clusters are designated as nl and 
n2, corresponding to the cluster designated as the first cluster and the cluster 
designated as the second cluster, respectively (step 1115). The designation of a 
cluster as the first or second cluster is arbitrary, and is driven by the need to identify 
the clusters with one of the following possible scenarios. If the size of both clusters is 
greater that two (i.e., each cluster contains at least two enterprise routers) (step 1 120), 
the process designated as Sub_2_2 is employed (step 11 30). If the size of the first 
cluster is at least 2, but the size of the second cluster is one (step 1 140), the process 
designated as Sub_2_l is employed (step 1 150). If both clusters have a size of one 
(step 1 160), the process designated as Sub_l_l is employed (step 1 170). It will be 
noted that the processes just referred to are designated using the minimum sizes for 
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the first and second clusters. If none of the aforementioned cases hold, and error 
condition exists and is indicated as such (step 1 180). 

The overall paradigm for apportioning delay components is now outlined 
using a formal representation. The process referred to herein as 
5 Apportion_delay components is one such description. 

BEGIN "Apportion delay components" 

/* Delay and jitter components are monitored between these two enterprise routers. */ 

10 

Specify a pair of routers, (B^^^, R(k2j)), kl ^ k2; 

Identify the corresponding clusters, and C,^; 

Identify the size, nl and n2, of the corresponding clusters; 

15 

/* Note that nl and n2 can be selected such that nl > n2 without loss of generality */ 
/* Three cases can be identified: */ 

/* The size of both the first and second clusters are greater than or equal to 2 */ 
20 If nl > 2, n2 > 2 then { 

Sub_2_2 0 

}; 

/* The size of the first cluster is greater than or equal to 2 */ 
25 /* The size of the second clusters is equal to 1 */ 
Ifnl >2,n2= 1 then { 
Sub_2_l 0 

}; 

30 /* The size of both the first and second clusters are equal to 1 */ 
Ifnl -l,n2-lthen{ 
Sub_l_l 0 
} 

35 END "Apportion delay components" 



Fig. 12 is a flow diagram illustrating the operations performed in the process 
of apportioning delay components when the size of both a first cluster and a second 
cluster are greater than or equal to two. The process begins with the identification of 
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the router pair (step 1200) and the definition of the measurements set (step 1210). 
Next, the delay-components vector is constructed (step 1220). The start time and 
measurement interval are then set (steps 1230 and 1240). While the measurement of 
delay and jitter has not been completed (step 1250), the following steps are carried 
5 out. For each entry in the measurements set (step 1255), and while within the time 
interval (step 1260), the average round-trip delay and delay-jitter between the given 
router pair is measured (step 1265). Once such measurements have been completed 
for each entry in the measurements set during the time interval, an estimate of round- 
trip delay, in the form of a rovmd-trip delay vector, is computed (step 1270). In a 
1 0 similar fashion, an estimate of round-trip jitter, in the form of a round-trip jitter 
vector, is computed (step 1280). Once these computations are completed, the start 
time may be incremented (step 1290) and the process repeated for the desired number 
of iterations. 

A method for apportioning delay components when the given pair of routers, 
15 (R(ki,i)5 ^^j))' ^oth have sizes larger than or equal to two (i.e., nl = | C^i j > 2, n2 = | 
I ^ 2), is now outlined using a formal representation. This process is referred to 
herein as Sub_2_2. 

BEGIN "Sub_ 2_2" 

20 

Identify routers, R^i_i+i), R(k2j+i); /* If i = nl , then let i+1 = 1 . 

Ifj =n2,thenletj+l = 1. */ 

/* Define the measurements set */ 
25 Q = { 

= (R(ki,i), R(ki,i^i)), 

^2 = (R(kl,i), R<k2j)), 
^3 ~ (^k2j)5 R(kl,i+1))5 
■^4 ^ (R(k2j)' -I^ITclj+l)): 
30 TC5 = (R(ki,i),R(k2J.l)) 

}; 

I* Note that H^^, ,^ = d^^,, ,^ + s^, ,^ */ 
/* Construct the delay-components vector */ 
■^^ X = [D(ki D(kj i+j-), C(]ji [^), D^j-), D(,^j+i)]; 
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to = start_time; 

dt = measurement_interval; 

While (Ifmished) { 
5 For each k =1 to 5, do { 

While within time interval (t,,, tg +dt) { 

Use a network monitoring tool to measure average round-trip 
delay, y^, and delay-jitter, y^, between the pair of routers, n^, in 
setQ. 

10 } 

}; 

Compute an estimate of link-level round-trip delay vector, x, using 



30 



8-[ 0.5 


0.5 


-0.5 
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0 


] [yi] 


[ 0.5 


-0.5 
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] [y2] 
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Compute an 


estimate of link-level round-trip jitter vei 


S = [ 0.25 
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] [y,'] 
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] [y/] 


[ 0.25 
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0.25 


0.25 


0.25 


] [y/] 


[ 0 
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0.25 


] [y4'] 


[ 0 


0.25 


0 


0.25 


0.25 


] [y/] 



where 2^ = [<y\,^^, a\,^^,^, cj" cr"^^^, CT\k2j+i)]; 

to = to+dt 
} 



END "Sub 2 2" 



Fig. 13 is a flow diagram illustrating the operations performed in the process 
of apportioning delay components when the size of a first cluster is greater than or 
equal to two and the size of a second cluster is equal to one. The process begins with 
the identification of a given router pair (step 1300) and the definition of the 
measurement set (step 1310). Next, a delay-components vector is constructed (step 
1320). The start time and measurement interval are then set (steps 1330 and 1340). 

-81 - 



Attorney Docket No. : M-79 1 5 US 



While further measurements are desired (step 1350), the following actions are 
performed. For each entry in the measurements set (step 1355), and while within the 
time interval (step 1360), the average round-trip delay and delay -jitter between the 
given router pair are measured (step 1365). Once each entry in the measurement set is 
5 analyzed during the given time interval, an estimate of round-trip delay, represented 
by a round-trip delay vector, is computed (step 1370). In a similar fashion, an 
estimate of round-trip delay-jitter is computed (step 1380). The start time is 
incremented (step 1390) and further iterations are performed, as desired. 

The method for apportioning delay components, when the size of a first cluster 
10 (nl) is greater than or equal to two and the size of a second cluster (n2) is equal to 

one, is now outlined using a formal representation. Thus, for the given pair of routers, 
(R(ki,i)' R(k2j))5 nl = I I > 2, n2 = I 1 = 1. This process is referred to herein as 
Sub_2_l. 
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BEGIN "Sub_2_l" 

Identify routers, Rg,i^i+i) /* If i = nl , then let i+1 = 1 . */ 

5 /* Define the measurements set */ 

^ = {^1 = (R(ki,i), R(ki,Ri)), = (R<ki,i)' R(k2o)), = (R(k2j> R(ki,i+i))}; 

/* Construct the delay-components vector */ 

10 

to = starttime; 

dt = measurement_interval; 

While (Ifmished) { 
1 5 For each k =1 to 3, do { 

While within time interval (to, to +dt) { 

Use a network monitoring tool to measure average round-trip 
delay, yK, and delay-jitter, y^, between the pair of routers, n^, in 
setQ. 

20 } 

}; 

Compute an estimate of link-level round-trip delay vector, x, using 

25 § = [ 0.5 0.5 -0.5 ] [yi] 

[ 0.5 -0.5 0.5 ] [yj 
[ -0.5 0.5 0.5 ] [y,]; 

Compute an estimate of link-level round-trip jitter vector, x, using 

30 

2 = [ 0.25 0.25 0.25 ] [y^^] 
[ 0.25 0.25 0.25 ] [y/ ] 
[ 0.25 0.25 0.25 ] [y/] 

35 where S"^ = [a\,^^, cj" (k,,;.,), a' (k,,ic2)+c^' (k2j)]; 

to = to+dt 
} 

40 END "Sub_ 2_1" 



Fig. 14 is a flow diagram illustrating the operations performed in the process 
of apportioning delay components when the size of both the first cluster and the 
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second cluster is equal to one. The process begins with the definition of the 
measurement set (step 1400) and the construction of the delay-components vector 
(step 1410). Next, the start time and measurement interval are set (steps 1420 and 
1430). While more measurements are to be taken (step 1440), and while within the 
5 time interval (step 1450), the average round-trip delay and delay-jitter between the 
given router pair are measured (step 1460). Once the measurements with in the time 
interval have been taken, an estimate of the round-trip delay, represented by the 
round-trip delay vector, is computed (step 1470). In similar fashion, an estimate of 
the round-trip delay-jitter is computed, resulting in a round-trip delay-jitter vector 
10 (step 1480). Once the afar mentioned computations have been made, the start time is 
incremented in order to take a crash set of measurements (step 1490). 

The method for apportioning delay components, when the size of a first cluster 
(nl) and the size of a second cluster (n2) are equal to one, is now outlined using a 
formal representation. Thus, for the given pair of routers, (Rq,i^^, R(k2j)), nl = | C^i | = 
15 1 , n2 = I €[,2 1 = 1 . This process is referred to herein as Sub_l_l . 

BEGIN "Sub_ 1_1" 

Q = {7Ti = (R(,,,),R,^,))}; 

20 

Construct the delay-components vector, = [D(ki,r)+^(ki,ia)+^(]aj)]l 

to = starttime; 

dt = measurement_interval; 

25 

While ([finished) { 

While within time uiterval (tg, to +dt) { 

Use a network monitoring tool to measure average round-trip delay, yj, 
30 and delay-jitter, yi, between the pair of routers, tTj = (R(ki,i), R(k2j)); 

Estimate link-level round-trip delay, D(ki,i)+C(ki k2)+D(^j), as yi; 
35 Estimate link-level round-trip jitter, ^(ki,i)+C(i,i^u)+^(k2,y), as y-^; 

to = to+dt 
} 
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END"Sub_l 1" 



Example scenarios 



Three examples are now considered to illustrate the processes described above. 
Example 1 

Consider the enterprise network depicted in Figs. lOA and lOB. Let the pair 
10 of routers be (R^ R(4 j^). Because =3 and = 2, sub_2_2 is applicable. As a 
result, 

n = {7ti, 712. 7x3= 7^4, (98a) 



^1 = R(i,2)) 

^2 = (R<1,1), R(4,l)) 

^3 = (R<4,1). ^^1,2)) 

^4 = (R(4,l)5 R(4,2)) 

^5 = ^4,2)) 



(98b) 
(98c) 
(98d) 
(98e) 
(98f) 



Further, the delay-components vector, = [D(u), D(j 2,, C(i 4), D(4 i^, D(4 2)]. An estimate 
of link-level roimd-trip delay vector, x, may be computed using 



[§D(1,1) ] 


[ 0.5 


0.5 


-0.5 


0 


0 


] [yi] 


(99a) 


[Sd(1,2) ] 


[ 0.5 


-0.5 


0.5 


0 


0 


] [72] 


(99b) 


[Sc(l,4) ] = 


[ -0.5 


0 


0.5 


-0.5 


0.5 


] [73] 


(99c) 


[Sd(4,1) ] 


[ 0 


0.5 


0 


0.5 


-0.5 


] [74] 


(99d) 


[Sd(4,2) ] 


[ 0 


-0.5 


0 


0.5 


0.5 


] [ys] 


(99e) 
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An estimate of link-level round-trip jitter vector, x, may be computed using 

[Cf^D(l,l)] [ 

['^^D(I,2)] [ 

5 [<,,4)]= [ 

b^D(4,l)] [ 

[f^^D(4,2)] [ 

Example 2 

10 Letting the pair of routers be (R(i^i), R(2,i)), and because Uj =3 and Uj = 1, sub_2_l is 
applicable. As a result, in a more compact form than that just given, 



0.25 


0.25 


0.25 


0 


0 


] [y.^] 


(lOOa) 


0.25 


0.25 


0.25 


0 


0 


] [72^] 


(100b) 


0.25 


0 


0.25 


0.25 


0.25 


] [y/] 


(100c) 


0 


0.25 


0 


0.25 


0.25 


] [y.'] 


(lOOd) 


0 


0.25 


0 


0.25 


0.25 


] [y/] 


(lOOe) 



Q = {Til = (R^j^,), R(i^2)), = (R(u)' R(2,i)), ^3 = (R(2,i), R(i.2))} (101) 

1 5 The delay-components vector, = [D(j j^, D(i 2), C(j 4)+D(2,i)]. An estimate of link-level 
round-trip delay vector, x, may be computed using 





[5d(i,]) ] 


[ 0.5 


0.5 


-0.5 


] [yi] 


(102a) 




[Sd(1,2) ] = 


[ 0.5 


-0.5 


0.5 


] [y2] 


(102b) 


20 


[^c(l,4)+D(2,l) ] 


[ -0.5 


0.5 


0.5 


] [y^^ 


(102c) 



An estimate of link-level round-trip jitter vector, x, may be computed using 

[cy%u)] [ 0.25 0.25 0.25 ][yi^](103a) 
25 bV2)]= [ 0.25 0.25 0.25 ] [y^'KlOSb) 

[<(UHD(2.i) ] [ 0.25 0.25 0.25 ][y3'](103c) 



It will be noted that it is not possible to apportion the delay or jitter between Cd 2) and 
0(2,1) separately since the number of routers in the second cluster is only 1. 

Example 3 

Let the pair of routers be (R^3,i), R(2,i))- Because =1 and rij = 1, sub_l_l is 
applicable. Because Q.= {n^= (R(3^i), R(2,i))}, no estimate of individual delay 
components is available. 
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V. Estimating Delay and Jitter in an Enterprise Network Coupled to a 
Backbone Having an Unknown Topology Using a Large Number of 
Router Pairs 

A method according to one embodiment of the present invention for 
5 computing the measiu-ements set, Q, for a given requirements set, O, has been 

described. If the size of the set O is P, then the method involves finding a maximal 
set of independent vectors fi-om a corresponding set of P row vectors. Such a 
computation may be unreasonably difficuh if P is large (e.g., 10000 or more). 
Therefore, a method according to an embodiment of the present invention is now 
1 0 described that allows the measurements set to be directly obtained. Measurements of 
delay and jitter between pairs of routers in this set may then be used to estimate delay 
and jitter between any set of pairs of routers and specified by <D. 

Fig. 1 5 is a flow diagram illustrating the operations performed in estimating 
delay/jitter between router pairs in a specified requirements set based on delay/jitter 

1 5 measurements between pairs of routers in a measurements set. This measurement set 
is directly obtained once the collection of routers that are covered by the requirements 
set is determined, and a model of the backbone network is computed. The process 
begins with the specification of the requirement set as a collection of router pairs (step 
1500). Next, the collection of enterprise routers for which delay and jitter estimates 

20 are required is identified (step 1510). The process of "form_clusters" (e.g., as 
previously described) is then used to: 

1 . determine a model for the enterprise network as one consisting of clusters of 
routers (step 1520), and to 

2. re-label the enterprise routers, as well as the requirements set (steps 1530 and 
25 1540). 

Since a value w(kl , k2) = 1 indicates that a delay over the corresponding 
virtual link of the backbone network is to be estimated, the matrix w is zeroed out 
because no such determinations have as yet been recorded in the matrix w (step 1545). 
Next, elements of the matrix signifying interconnection between clusters are set to one 
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if the delay experienced over the corresponding backbone link needs to be estimated 
(step 1550). The vector containing delay components is then identified (step 1555). 

At this juncture, the process that computes the measurements sets is called, in 
order to identify the measurements set and to create a record of equations that allows 
5 the estimation of delay and jitter (1560). Next, the start time and measurement 

interval are set (steps 1565 and 1570). While further estimations of delay and jitter 
remain to be made (step 1 575), the following steps are carried out. First, the average 
round-trip delay/jitter for every router pair is measured during the time interval (step 
1580). From these measurements, delay and jitter for each delay component is 
1 0 estimated (step 1 585). Also estimated is the round-trip and round-jitter for each router 
pair in O (step 1590). If fiirther measurements are desired, the start time is 
incremented and the process repeated (step 1595). 

The preceding method for computing the measurements set for a given 
requirements set when a maximal set of independent vectors from a corresponding set 
15 of row vectors is to be foimd is now outlined using a formal representation. 

BEGIN 



/* Specify the requirements set as a collection of router pairs */ 
20 /* Note that the following router pairs may not be, for example, (R^i Pj^) */ 

/* Further, without loss of generality, note that it can be assumed that i < j */ 

/* (This is valid because round-trip delay and jitter are examined, */ 

/* and so tTk = (Ri, Rj)} = (Rj, R^) can be assumed) */ 



25 ^={7rK = (R^,R3),K=l,2, ...,P}; 

/* From the given requirements set, O, identify */ 
/* the collection of enterprise routers, 5R, for which */ 
/* delay and jitter estimates are required */ 

30 

91 - u Ri; /* such that for some R^, (R;, R^) € O or (R^, R^) e (D */ 



/* Given dl, use "form_clusters" to: */ 

/* (1) Determine a model for the enterprise network as one having */ 

35 /* clusters of routers, Ck= {R(k,,).i= 1=2, ...,nj,k= 1,2, ...,m, */ 

/* where nk= | | ' */ 
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/* (2) Re-label the requirements set, O = (tTk = (I^ki,i). R(k2j))} */ 
form_clusters (); 

/* Re-order elements of O */ 
For all K = 1 to P do { 

Relabel ti^ = (R,, R^) as tz^ = (R^i,o,R(k2j)) 

} 

/* Identify interconnection between clusters using virtual service-provider links. This 
is specified as a m x m matrix, co(kl, k2). Note that co(kl,k2) = 1 indicates that delay 
over link Xq^^^^ is required to be estimated, by definition. */ 

/* Zero out the matrix co */ 
For all kl = 1,2, ...,mdo { 

For all k2= 1,2, ... mdo{ 
co(kl,k2) = 0 

} 

} 

/* Set matrix elements to one if the delay over the corresponding backbone link */ 
/* needs to be estimated */ 
For all {ttk = (R(ki,i), R<k2j))} e O do { 

co(kl,k2)= 1 

} 

/* Identify the vector of delay components */ 

{D(ki3,k- 1,2, ...,m,i = 1,2, ... n^,} u {C(k,,k2), kl,k2 = 1,2, ...,m, co(kl,k2) = 1} 



/* Call compute_measurement_set to: */ 

/* Identify the measurements set, Q = {ti'k = (R^i,i), R(k2j), ^=1,2, ...,Q} */ 

/* Create record of eqns allowing est. of delay/jitter) of D(kl ,i) or c(kl ,k2): */ 

/* est_delay() = fimction(yK, . . . , est_delay(D(ki4)), . . . ) */ 

/* estJitterO ^ function(yK, . . . , estJitter(D(ki,i)), . . .) */ 



compute_measurement_set( ); 

to = start_time; 

dt = measurement_interval; 

While ([finished) { 

Measure the average round-trip delay, yK, and delay-jitter, Yk, over time 
interval [to, to+dt] for every pair of routers, tt k = (R(ki,i), R(k2j)) ^ using a 
network monitoring tool; 
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/* Estimate delay and jitter for each delay component */ 
For all k= 1,2, ...,mdo { 

For all i= 1,2, ...,n^do { 

evaluate est_delay(D([, j^); 

evaluate estJitter(D(k^i)) 

} 

} 

For all kl = 1,2, ...,mdo { 

Forallk2 = kl+l,k2+2, ...,mdo { 
ifco(kl,k2) = lthen{ 

evaluate est_delay(C(ki,k2)); 
evaluate estJitter(C(i,i ,^)) 
} 

} 

} 

/* For each router pair in O, estimate rovmd-trip delay, Ak, and round-trip jitter, cTk */ 
For each ttk = (R(ki,i), R(k2j)) e O do { 

If(((R(ku), Roc2j)) = 7r',i)e n)then{ 
Ak = yKi; 

} 

else { 

ifkl =k2then { 

Ak = est_delay(D(ki + est_delay(D(i^j)); 

cTk = V (estJitter(D(ki,i))' + estJitter(D(k2j))') 
} 



est_delay(D(ki,i)) + 
est_delay(C(ki^k2)) + 
est_Delay(D(k2j)); 

V ( estJitter(D(k,,,))' + 
estjitter(c^i k2))2 + 
estJitter(D(^j))^) 



to = to+dt 
} 
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Fig. 1 6 is a flow diagram illustrating the operations performed in computing 
the measurements set needed in the preceding method. It is assumed that the given 
network has been described in terms of clusters and an interconnection matrix, using a 
method of clustering such as that discussed previously. The process begins with the 
initialization of the measurements set (step 1600). For all clusters (defined by kl = 1, 
m) (step 1604), the following steps are performed. 

First, the size of the given cluster is examined (step 1606). If the given cluster 
has two or more enterprise routers, a second cluster (identified by k2) is identified, 
such that delay/jitter over the backbone link (virtual link) between the two clusters (or, 
alternatively, backbone routers) must be obtained (i.e., such that (o(kl,k2) =1 or 
co(k2,kl) = 1) (step 1608). Having identified this second cluster, the procedure 
identifies three specific routers and adds the corresponding pairs of routers to the 
measurements set (steps 1610, 1612 and 1614). Expressions representing estimated 
delay over links coimecting enterprise routers or over the virtual backbone links are 
then recorded (step 1616). 

The rest of the enterprise routers in the given cluster are then analyzed (step 
1620). First, a router pair corresponding every third, fourth, router in the cluster 
ciirrent is added to the measurements set (step 1622). Next, expressions representing 
estimated delay and jitter between associated backbone and enterprise routers is 
recorded (steps 1624 and 1626). This completes the scenario in which the given 
cluster's size is greater than or equal to two. 

If the given cluster's size is equal to one (step 1606), the following actions are 
performed. A delay estimate of zero is recorded, indicating that such an estimate is 
not available (step 1628). In similar fashion, a jitter estimate of zero is recorded, 
likewise indicating that such an estimate is not available (step 1630). This completes 
this portion of the analysis. 

Once the addition of router pairs to the measurements set and the recording of 
delay and jitter expressions are completed, the inter-cluster router pairs (and 
associated delay and jitter) are determined. For pairs of clusters consisting of a first 
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cluster (one of all clusters) and a second cluster (one of all remaining clusters) the 
following actions are performed (steps 1632 and 1634). If delay between backbone 
routers corresponding to the two clusters must be estimated (step 1636), then a pair of 
enterprise routers representing the two clusters is added to the measurements set (step 
5 1638). Expressions representing the estimated delay (step 1640) and estimated jitter 
(step 1642) associated with the given pair of backbone routers are recorded. Finally, 
the size of the measurements set is stored (step 1644). This completes computation of 
the measurements set. 

The preceding method for computing the measurements set is now outlined 
1 0 using a formal representation. For a given network described in terms of clusters: 

Ck = i = 1, 2, . . nj, k = 1, 2, . . ., m, where | C, | (104) 

and an interconnection matrix: 

15 

(B(kl,k2),kl,k2= 1,2, ...,m (105) 



the measurements set can be computed as follows: 



20 BEGIN "compute measurements set" 

Q={} 

K = 0 

For all kl = 1,2, ...,mdo { 
25 Ifn5,i>2then{ 



/* Q = measurements set */ 

/* K identifies the measurements, yK2 */ 



/* It is assumed that such a k2 exists */ 

identify any k2, such that co(kl,k2) =1 or a>(k2,kl) = 1; 

30 k1=k+1; 

add_to_set(7rKi = (R(ki.i)' ^U2)l 
k2 = K +2; 

add_to_set(7tK2 = (R(ku), R(k2,i)), ^); 
k3 =K+3; 

35 add_to_set(7iK3 = '^i,2)X ^); 

K = K+3; 

/* One method to record an expression is to use a binary evaluation tree */ 
-92- 
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rec_eqn[est_delay(D(ki,i)) = 0.5 y^i + 0.5 y^^ - 0.5 y^,]; 
rec_eqn[est_delay(D(k,,2)) = 0.5 y^, - 0.5 + 0.5 yK3]; 

/* An estimate of jitter is similarly derived */ 
5 rec_eqn[estJitter(D(ku)) = V (0.25 y^Ki + 0.25 y^.^ + 0.25 /ks)]; 

rec_eqn[estJitter(D(ki,2)) = V(0.25 /ki + 0.25 / k2 + 0.25 y^^j)]; 

For all i = 3, 4, . . ., n^i do { 
K = K+1; 

1 0 add_to_set(7r K = (I^u). ^u)), ^Y, 

rec_eqn[est_delay(D(ki i)) = Yk - est_delay(D(ki,i))]; 
rec_eqn[estJitter(D(ki,i)) = V(y\ + estJitter(D(k, i))'); 
} 

} 

15 else { 

/* A delay estimate of 0 signifies that estimate is not available */ 
rec_eqn[est_delay(D(kij)) = 0]; 
rec_eqn[estJitter(D(ki i)) = 0]; 

} 

20 } 

For all kl = 1,2, ...,mdo { 

Forallk2 = kl+l,kl+2, ...,mdo { 
ifco(kl,k2)= 1 then { 
25 K = K+1; 

add_to_set(7rK - (R^i,i), B^^i-,), O); 

rec_eqn[est_delay(C(ki,k2)) = Yk - est_delay(D(ki,i)) - est_delay(D(k2,i))]; 
rec_eqn[estJitter(C(ki,k2)) = V(f k +estJitter(D(ki,i))' +estJitter(D(k2 1))')] 
} 

30 } 

} 

Q = k /* Store the size of the measurements set */ 

END "compute measurements set" 



Using the foregoing method, an estimate of all (or nearly all) delay 
components is available, depending on the topologies involved. Thus, it is possible to 
apportion end-to-end delay among different links that constitute the given path. It 
will be noted that the measurements set, fi, may or may not be a sub-set of the 
40 requirements set, O. The total number of measurements required is at most: 
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(ni-l+n2-l+ ... n„-l) + 2m + m(m-l)/2 (105) 

This bound is achieved when every cluster has at least 2 routers, and delay/jitter 
estimates are required between every pair of clusters. Moreover, the above method 
suggests that delay and jitter measurements between = (R(ki,i), R(k2,i)) may be made 
more than once. In fact, this may be rectified during implementation by checking 
whether a delay/jitter measurement between = (R(kij), R^^d) is already available. In 
such a case, the pre-existing measurement is used. 

Exemplary Enterprise Network 

As an example, the enterprise network of Fig. lOB may again be considered, 
with the requirements set: 

<D = {(R(,,,), R^2,l)), (R(l,2> R(2,l)). (R<U)' R(2,l)), 
(R<1,1)5 R(3,l))5 (R(l,2)5 R(3,l))5 (R(l,3)5 R(3,l))' 
R(4,l))' (1^1,2)5 R(4,l))j (R(l,3)' ^4,1))^ 
R(4,2)). (R(l,2). R(4,2)). (R(l,3)' R(4,2))} 

The collection of routers is = {R^i i), R^i 2), R(i^3), R^2,i). R(4,i), R(4,2)}- Further, the 
matrix, oo is 





K2=l 


K2=2 


K2=3 


K2=4 


Kl=l 




1 


1 


1 


Kl=2 










Kl=3 










Kl=4 











The delay-component vector is given as 



[0(1,3) 
P(2,l) 
P(3,,) 

P(4,2) 
[^^(1,2) 
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[C(l,4) ] 

Using the subroutine "compute_measurements_set," the measurements set is 
obtained as follows: 

(i) ^={} 

(ii) Let kl=l: because n^ = 3, Q <e Q u {(R^,^,^, R^^^^)), (R(i,i), R(2,.)), (R(2,i), R(i.2)), 

(R<1,1)' R(l,3))} 

(iii) Let kl=2: because n2 = 1, there is no change in Q. , 

(iv) Let kl=3: because = 1, there is no change in Q. 

(V) Let kl=4, because n, = 2, Q = Q u {(R^^,), R^,^)), (R<4,i), R(u)), (R<ui), R(4,2))} 

(vi) Let kl=l, k2=2, because co(kl,k2)=l , Q = Q u {(Rd^), R(2,i))} 

(vii) Let kl=l, k2=3, because a)(kl,k2)=l, Q = Q u {(R(i,i), R(3^i))} 

(viii) Let kl-1, k2=4, because (D(kl,k2)=l, Q = Q u {(R(i,i), R<4,i))}. 

As a result, 

^ ^ ((^1,1)' ^1,2))^ ^2,1))^ (^2,l)y R(l,2))' CR(1,I)' ^^1,3))= 

(R(4,l)' I^4,2))5 (R<4,1)» 1^1,1))' (^1,1)' I^4,2))5 1^3,1))} 
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20 



and the estimates of various delay components may be obtained from the 
corresponding measurements, y;, i = 1, 2, 8, as follows: 

est_delay(D(i = 0.5 yi + 0.5 - 0.5 yj 
est_delay(D(i'2)) = 0.5 y, - 0.5 y^ + 0.5 y3 
est_delay(D(i 3)) = y^ - est_delay(D(i i^) 

est_delay(D(2j)) = 0 /* (no estimate is available) */ 

est_delay(D(3 = 0 /* (no estimate is available) */ 

est_delay(D(4|i)) = 0.5 y, + 0.5 y^ - 0.5 y, 

est_delay(D(4,2)) = 0.5 y, - 0.5 y, + 0.5 y^ 

est_delay(C(i 2)) = - est_delay(D(i d) - est_delay(D(2,i))] 

est_delay(C(i 3)) - yg - est_delay(D(,'i)) - est_delay(D(3|i))] 

est_delay(C(i^4^) = y^ - est_delay(D(, d) - est_delay(D(4 ,))] 

These estimates may now be combined to estimate the delay between any pair 
of enterprise routers. For instance, the delay between R^i 3) and R(4 2) may be estimated 
to be 

est_delay(D(i 3)) + est_delay(C(i 4^) + est_delay(D(4 2)) 

The equations for estimating jitter are similar. 

While particular embodiments of the present invention have been shown and 
described, it will be obvious to those skilled in the art that, based upon the teachings 
herein, changes and modifications may be made without departing from this invention 
and its broader aspects and, therefore, the appended claims are to encompass within 
their scope all such changes and modifications as are within the true spirit and scope 
of this invention. Furthermore, it is to be understood that the invention is solely 
defined by the appended claims. 
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WHAT IS CLAIMED IS: 



1 1 . A method of determining a network performance metric in a network, 

2 wherein said network comprises a plurality of network elements and each one of said 

3 network elements is coupled to at least one other of said network elements by at least 

4 one of a plurality of links, comprising: 

5 measuring a measured network performance metric between a first network 

6 element and a second network element of a network element pair in a 

7 first set of network element pairs, wherein 

8 said first network element and said second network element of said 

9 network element pair in said first set of network element pairs 

10 is included in said network elements, and 

1 1 said measured network performance metric is such that a computed 

1 2 network performance metric between a first network element 

1 3 and a second network element of said network elements can be 

1 4 computed using said measured network performance metric. 

1 2. The method of claim 1, wherein 

2 said computed network performance metric can be computed between 

3 a first network element and a second network element of a 

4 network element pair in a second set of network element pairs 

5 using said measured network performance metric, and 

6 said first network element and said second network element of said 

7 network element pair in said second set of network element 

8 pairs is included in said network elements. 

1 3. The method of claim 2, further comprising: 

2 computing said computed network performance metric between said first 

3 network element and said second network element of said network 

4 element pair in said second set of network element pairs, wherein 

5 said second set of network element pairs are included in said network 

6 elements, and 
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7 said first set of network element pairs is included in a second set of 

8 network element pairs. 

1 4. The method of claim 3, further comprising: 

2 identifying pairs of said network elements as being in said second set of 

3 network element pairs; and 

4 identifying pairs of said network elements in said second set of network 

5 element pairs as being in said first set of network element pairs. 

1 5. The method of claim 4, further comprising: 

2 generating a first matrix using said second set of network element pairs, 

3 wherein 

4 each row in said first matrix corresponds to a corresponding network 

5 element pair in said second set of network element pairs, 

6 said first matrix comprises independent rows and non-independent 

7 rows, 

8 said first set of network element pairs contains independent network 

9 element pairs in said second set of network element pairs, and 

10 each one of said independent pairs of network element corresponds to a 

1 1 one of said independent rows of said first matrix. 

1 6. The method of claim 5, wherein said measured network performance 

2 metric is measured between a first network element and a second network element of 

3 each network element pair in said first set of network element pairs. 

1 7. The method of claim 5, said method further comprising: 

2 computing a number, wherein said number is equal to a rank of said first 

3 matrix; 

4 determining if a first said nximber of rows of said first matrix are independent; 

5 and 
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6 if said first said number of said rows of said first matrix are not independent, 

7 re-arranging said rows of said first matrix such that said first said 

8 number of said rows of said first matrix are independent. 

1 8. The method of claim 7, further comprising: 

2 identifying a maximal set of independent rows of said first matrix based on 

3 said number. 

1 9. The method of claim 7, wherein said rearranging further comprises: 

2 re-arranging said pairs of said network elements in said second set of network 

3 element pairs such that said correspondence between each row of said 

4 first matrix and said corresponding network element pair in said second 

5 set of network element pairs is maintained. 

1 10. The method of claim 9, wherein said forming said first set of network 

2 element pairs comprises: 

3 copying a first said number of pairs of said network elements in said second 

4 set of network element pairs into said first set of network element 

5 pairs. 

1 11. The method of claim 5, wherein 

2 said network element pair in said second set of network element pairs is a 

3 remaining network element pair in said second set of network element 

4 pairs, and 

5 said remaining network element pair corresponds to a non-independent row of 

6 said first matrix 

1 12. The method of claim 6, wherein said computing said computed 

2 network performance metric between said first network element and said second 

3 network element of said remaining network element pair comprises: 

4 forming a second matrix, wherein 
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5 each row of said second matrix corresponds to a corresponding one of 

6 said non-independent rows of said first matrix, and 

7 said each row of said second matrix is such that said corresponding one 

8 of said non-independent rows of said first matrix can be 

9 expressed in terms of said independent rows using said each 

10 row of said second matrix; 

1 1 organizing said measured network performance metrics into a vector; and 

12 computing said computed network performance metric between said first 

1 3 network element and said second network element of said remaining 

14 network element pair by multiplying said vector by a row of said 

15 second matrix corresponding to said remaining network element pair. 

1 13. The method of claim 6, wherein said computing said computed 

2 network performance metric between said first network element and said second 

3 network element of said remaining network element pair comprises: 

4 creating a vector equivalent to said non-independent row of said first matrix by 

5 combining a plurality of said independent rows of said first matrix; and 

6 computing said computed network performance metric by combining a 

7 measured network performance metric of each network element pair of 

8 said first set of network element pairs corresponding to one of said 

9 plurality of said independent rows of said first matrix. 

1 14. A method of determining a network performance metric in a network, 

2 wherein said network comprises a plurality of network elements and each one of said 

3 network elements is coupled to at least one other of said network elements by at least 

4 one of a plurality of links, comprising: 

5 identifying pairs of said network elements as being in a first set of network 

6 element pairs; 

7 generating a first matrix from said first set of network element pairs, wherein 

8 each row in said first matrix corresponds to a corresponding network 

9 element pair in said first set of network element pairs, and 
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1 0 said first matrix comprises independent rows and non-independent 

1 1 rows; 

12 forming a second set of network element pairs, wherein 

1 3 said second set of network element pairs contains independent network 

14 element pairs in said first set of network element pairs, and 

1 5 each one of said independent pairs of network element corresponds to a 

1 6 one of said independent rows of said first matrix; 

1 7 measuring a measured network performance metric between a first network 

1 8 element and a second network element of each network element pair in 

1 9 said second set of network element pairs; and 

20 computing a computed network performance metric between a first network 

21 element and a second network element of a remaining network element 
- 22 pair in said first set of network element pairs using at least one of said 

23 measured network performance metrics, wherein said remaining 

24 network element pair corresponds to a non-independent row of said 

25 first matrix. 

1 15. The method of claim 14, wherein said first set of petwork element pairs 

2 is a requirements set. 

: 1 16. The method of claim 1 5, wherein said second set of network element 

2 pairs is a measurements set. 

1 17. The method of claim 1 6, wherein each one of said network elements is 

2 a router. 

1 18. The method of claim 1 4, wherein said first matrix is a matrix F. 
2 

1 1 9. The method of claim 14, said method further comprising: 

2 computing a number, wherein said number is equal to a rank of said first 

3 matrix; 
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4 determining if a first said number of rows of said first matrix are independent; 

5 and 

6 if said first said number of said rows of said first matrix are not independent, 

7 re-arranging said rows of said first matrix sucli that said first said 

8 number of said rows of said first matrix are independent. 

1 20. The method of claim 1 9, further comprising: 

2 identifying a maximal set of independent rows of said first matrix based on 

3 said number. 

1 21. The method of claim 1 9, wherein said rearranging fiarther comprises : 

2 re-arranging said pairs of said network elements in said first set of network 

3 element pairs such that said correspondence between each row of said 

4 first matrix and said corresponding network element pair in said first 

5 set of network element pairs is maintained. 

1 22. The method ofclaim 21, wherein said forming said second set of 

2 network element pairs comprises: 

3 copying a first said number of pairs of said network elements in said first set 

4 of network element pairs into said second set of network element pairs. 

1 23. The method of claim 14, wherein said computing said computed 

2 network performance metric between said first network element and said second 

3 network element of said remaining network element pair comprises: 

4 forming a second matrix, wherein 

5 each row of said second matrix corresponds to a corresponding one of 

6 said non-independent rows of said first matrix, and 

7 said each row of said second matrix is such that said corresponding one 

8 of said non-independent rows of said first matrix can be 

9 expressed in terms of said independent rows using said each 

10 row of said second matrix; 

1 1 organizing said measured network performance metrics into a vector; and 
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12 computing said computed network performance metric between said first 

13 network element and said second network element of said remaining 

1 4 network element pair by multiplying said vector by a row of said 

1 5 second matrix corresponding to said remaining network element pair. 

1 24. The method of claim 23, wherein said second matrix is a matrix A. 

1 25. The method of claim 14, wherein said computing said computed 

2 network performance metric between said first network element and said second 

3 network element of said remaining network element pair comprises: 

4 creating a vector equivalent to said non-independent row of said first matrix by 

5 combining a plurality of said independent rows of said first matrix; and 

6 computing said computed network performance metric by combining a 

7 measured network performance metric of each network element pair of 

8 said second set of network element pairs corresponding to one of said 

9 plurality of said independent rows of said first matrix. 

1 26. The method of claim 14, wherein each one of said network elements is 

2 a router. 

1 27. A computer system comprising: 

2 a processor; 

3 a network interface, coupled to said processor and to a network, wherein said 

4 network comprises a plurality of network elements and each one of 

5 said network elements is coupled to at least one other of said network 

6 elements by at least one of a plurality of links; 

7 computer readable medium coupled to said processor; and 

8 computer code, encoded in said computer readable medium, configured to 

9 cause said processor to: 

10 identify pairs of said network elements as being in a first set of 

1 1 network element pairs; 
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12 generate a first matrix from said first set of network element pairs, 

13 wherein 

1 4 each row in said first matrix corresponds to a corresponding 

1 5 network element pair in said first set of network element 

16 pairs, and 

17 said first matrix comprises independent rows and non- 

1 8 independent rows; 

1 9 form a second set of network element pairs, wherein 

20 said second set of network element pairs contains independent 

2 1 network element pairs in said first set of network 

22 element pairs, and 

23 each one of said independent pairs of network element 

24 corresponds to a one of said independent rows of said 

25 first matrix; 

26 measure a measured network performance metric between a first 

27 network element and a second network element of each 

28 network element pair in said second set of network element 
3 29 pairs; and 

30 compute a computed network performance metric between a first 

3 1 network element and a second network element of a remaining 

32 network element pair in said first set of network element pairs 

33 using at least one of said measured network performance 

34 metrics, wherein said remaining network element pair 

35 corresponds to a non-independent row of said first matrix. 

1 28. The computer system of claim 27, wherein said first set of network 

2 element pairs is a requirements set. 

1 29. The computer system of claim 28, wherein said second set of network 

2 element pairs is a measurements set. 
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1 30. The computer system of claim 29, wherein each one of said network 

2 elements is a router. 

1 31. The computer system of claim 27, wherein said first matrix is a matrix 

2 F. 

1 32. The computer system of claim 27, wherein said computer code is 

2 fiirther configured to cause said processor to: 

3 compute a number, wherein said number is equal to a rank of said first matrix; 

4 determine if a first said number of rows of said first matrix are independent; 

5 and 

6 if said first said number of said rows of said first matrix are not independent, 

7 re-arrange said rows of said first matrix such that said first said number 

8 of said rows of said first matrix are independent. 

1 33. The computer system of claim 32, wherein said computer code is 

2 further configured to cause said processor to: 

3 identify a maximal set of independent rows of said first matrix based on said 

4 number. 

1 34. The computer system of claim 32, wherein said computer code 

2 configured to cause said processor to re-arrange said rows of said first matrix such 

3 that said first said number of said rows of said first matrix are independent, if said first 

4 said number of said rows of said first matrix are not independent, is further configured 

5 to cause said processor to: 

6 re-arrange said pairs of said network elements in said first set of network 

7 element pairs such that said correspondence between each row of said 

8 first matrix and said corresponding network element pair in said first 

9 set of network element pairs is maintained. 
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1 35. The computer system of claim 34, wherein said computer code 

2 configured to cause said processor to form said second set of network element pairs is 

3 configured to cause said processor to: 

4 copy a first said number of pairs of said network elements in said first set of 

5 network element pairs into said second set of network element pairs. 

1 36. The computer system of claim 27, wherein said computer code 

2 configured to cause said processor to compute said computed network performance 

3 metric between said first network element and said second network element of said 

4 remaining network element pair is configured to cause said processor to: 

5 form a second matrix, wherein 

6 each row of said second matrix corresponds to a corresponding one of 

7 said non-independent rows of said first matrix, and 

8 said each row of said second matrix is such that said corresponding one 

9 of said non-independent rows of said first matrix can be 

1 0 expressed in terms of said independent rows using said each 

1 1 row of said second matrix; 

12 organize said measured network performance metrics into a vector; and 

13 compute said computed network performance metric between said first 

1 4 network element and said second network element of said remaining 

1 5 network element pair by multiplying said vector by a row of said 

1 6 second matrix corresponding to said remaining network element pair. 

1 37. The computer system of claim 36, wherein said second matrix is a 

2 matrix A. 
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1 38. The computer system of claim 27, wherein said computer code 

2 configured to cause said processor to compute said computed network performance 

3 metric between said first network element and said second network element of said 

4 remaining network element pair is further configured to configured to cause said 

5 processor to: 

6 create a vector equivalent to said non-independent row of said first matrix by 

7 combining a plurality of said independent rows of said first matrix; and 

8 compute said computed network performance metric by combining a 

9 measured network performance metric of each network element pair of 

1 0 said second set of network element pairs corresponding to one of said 

1 1 plurality of said independent rows of said first matrix. 

1 39. The computer system of claim 27, wherein each one of said network 

2 elements is a router. 

1 40. A computer program product encoded in computer readable media, 

2 said computer program product comprising: 

3 a first set of instructions, executable on a computer system, configured to 

4 identify pairs of said network elements as being in a first set of 

5 network element pairs, wherein 

6 said computer system is coupled to a network, wherein said network 

7 comprises a plurality of network elements and each one of said 

8 network elements is coupled to at least one other of said 

9 network elements by at least one of a plurality of links; 

10 a second set of instructions, executable on said computer system, configured to 

1 1 generate a first matrix from said first set of network element pairs, 

12 wherein 

13 each row in said first matrix corresponds to a corresponding network 

14 element pair in said first set of network element pairs, and 

1 5 said first matrix comprises independent rows and non-independent 

1 6 rows; 
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17 a third set of instructions, executable on said computer system, configured to 

1 8 form a second set of network element pairs, wherein 

1 9 said second set of network element pairs contains independent network 

20 element pairs in said first set of network element pairs, and 

2 1 each one of said independent pairs of network element corresponds to a 

22 one of said independent rows of said first matrix; 

23 a fourth set of instructions, executable on said computer system, configured to 

24 measure a measured network performance metric between a first 

25 network element and a second network element of each network 

26 element pair in said second set of network element pairs; and 

27 a fifth set of instructions, executable on said computer system, configured to 

28 compute a computed network performance metric between a first 

29 network element and a second network element of a remaining network 

30 element pair in said first set of network element pairs using at least one 

31 of said measured network performance metrics, wherein said remaining 

32 network element pair corresponds to a non-independent row of said 

33 first matrix. 

1 41. The computer program product of claim 40, wherein said first set of 

2 network element pairs is a requirements set. 

1 42. The computer program product of claim 41 , wherein said second set of 

2 network element pairs is a measurements set. 

1 43 . The computer program product of claim 42, wherein each one of said 

2 network elements is a router. 

1 44. The computer program product of claim 40, wherein said first matrix is 

2 a matrix F. 
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1 45. Thecomputerprogramproductof claim 44, further comprising: 

2 a sixth set of instructions, executable on said computer system, configured to 

3 compute a number, wherein said number is equal to a rank of said first 

4 matrix; 

5 a seventh set of instructions, executable on said computer system, configured 

6 to determine if a first said number of rows of said first matrix are 

7 independent; and 

8 an eighth set of instructions, executable on said computer system, configured 

9 to re-arrange said rows of said first matrix such that said first said 

1 0 number of said rows of said first matrix are independent, if said first 

1 1 said number of said rows of said first matrix are not independent. 

1 46. The computer program product of claim 45, further comprising: 

2 a ninth set of instructions, executable on said computer system, configured to 

3 identify a maximal set of independent rows of said first matrix based 

4 on said number. 

1 47. The computer program product of claim 45, wherein said eighth set of 

2 instructions comprises: 

3 a first sub-set of instructions, executable on said computer system, configured 

4 to re-arrange said pairs of said network elements in said first set of 

5 network element pairs such that said correspondence between each row 

6 of said first matrix and said corresponding network element pair in said 

7 first set of network element pairs is maintained. 

1 48. The computer program product of claim 47, wherein said third set of 

2 instructions comprises: 

3 a second sub-set of instructions, executable on said computer system, 

4 configured to copy a first said number of pairs of said network 

5 elements in said first set of network element pairs into said second set 

6 of network element pairs. 
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1 49. The computer program product of claim 40, wherein said fifth set of 

2 instructions comprises: 

3 a first sub-set of instructions, executable on said computer system, configured 

4 to form a second matrix, wherein 

5 each row of said second matrix corresponds to a corresponding one of 

6 said non-independent rows of said first matrix, and 

7 said each row of said second matrix is such that said corresponding one 

8 of said non-independent rows of said first matrix can be 

9 expressed in terms of said independent rows using said each 

1 0 row of said second matrix; 

11 a second sub-set of instructions, executable on said computer system, 

12 configured to organize said measured network performance metrics 

1 3 into a vector; and 

14 an third sub-set of instructions, executable on said computer system, 

1 5 configxired to compute said computed network performance metric 

1 6 between said first network element and said second network element of 

17 said remaining network element pair by multiplying said vector by a 

1 8 row of said second matrix corresponding to said remaining network 

19 element pair. 

1 50. The computer program product of claim 49, wherein said second 

2 matrix is a matrix A. 

1 51. The computer program product of claim 40, wherein said fifth set of 



2 instructions comprises: 



3 a first sub-set of instructions, executable on said computer system, configured 

4 to create a vector equivalent to said non-independent row of said first 

5 matrix by combining a plurality of said independent rows of said first 

6 matrix; and 

7 a second-subset of instructions, executable on said computer system, 

8 configured to compute said computed network performance metric by 
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9 combining a measured network performance metric of each network 

1 0 element pair of said second set of network element pairs corresponding 

1 1 to one of said plurality of said independent rows of said first matrix. 

1 52. The computer program product of claim 40, wherein each one of said 

2 network elements is a router. 

1 53. A computer system comprising: 

2 a network interface, coupled to said processor and to a network, wherein said 

3 network comprises a plurality of network elements and each one of 

4 said network elements is coupled to at least one other of said network 

5 elements by at least one of a plurality of links; 

6 means for identifying pairs of said network elements as being in a first set of 

7 network element pairs; 

8 means for generating a first matrix from said first set of network element pairs, 

9 wherein 

10 each row in said first matrix corresponds to a corresponding network 

1 1 element pair in said first set of network element pairs, and 

12 said first matrix comprises independent rows and non-independent 

1 3 rows; 

14 means for forming a second set of network element pairs, wherein 

15 said second set of network element pairs contains independent network 

1 6 element pairs in said first set of network element pairs, and 

1 "7 each one of said independent pairs of network element corresponds to a 

1 8 one of said independent rows of said first matrix; 

1 9 means for measuring a measured network performance metric between a first 

20 network element and a second network element of each network 

21 element pair in said second set of network element pairs; and 

22 means for computing a computed network performance metric between a first 

23 network element and a second network element of a remaining network 

24 element pair in said first set of network element pairs using at least one 

25 of said measured network performance metrics, wherein said remaining 
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26 network element pair corresponds to a non-independent row of said 

27 first matrix. 

1 54. The computer system of claim 53, further comprising: 

2 compute a number, wherein said number is equal to a rank of said first matrix; 

3 means for determining if a first said number of rows of said first matrix are 

4 independent; and 

5 means for re-arranging said rows of said first matrix such that said first said 

6 number of said rows of said first matrix are independent, if said first 

7 said number of said rows of said first matrix are not independent. 

1 55. The computer system of claim 54, wherein said computer code is 

2 further configured to cause said processor to: 

3 means for identifying a maximal set of independent rows of said first matrix 

4 based on said number. 

1 56. The computer system of claim 54, wherein said means for re-arranging 

2 said rows of said first matrix such that said first said number of said rows of said first 

3 matrix are independent, if said first said number of said rows of said first matrix are 

4 not independent, further comprises: 

5 means for re-arranging said pairs of said network elements in said first set of 

6 network element pairs such that said correspondence between each row 

7 of said first matrix and said corresponding network element pair in said 

8 first set of network element pairs is maintained. 

1 57. The computer system of claim 56, wherein said means for forming said 

2 second set of network element pairs further comprises: 

3 means for copying a first said number of pairs of said network elements in said 

4 first set of network element pairs into said second set of network 

5 element pairs. 
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1 58. The computer system of claim 53, wherein said means for computing 

2 said computed network performance metric between said first network element and 

3 said second network element of said remaining network element pair further 

4 comprises: 

5 means for forming a second matrix, wherein 

6 each row of said second matrix corresponds to a corresponding one of 

7 said non-independent rows of said first matrix, and 

8 said each row of said second matrix is such that said corresponding one 

9 of said non-independent rows of said first matrix can be 

0 expressed in terms of said independent rows using said each 



1 1 row of said second matrix; 

12 means for organizing said measured network performance metrics into a 

1 3 vector; and 

14 means for computing said computed network performance metric between said 

1 5 first network element and said second network element of said 

1 6 remaining network element pair by multiplying said vector by a row of 

1 7 said second matrix corresponding to said remaining network element 

1 8 pair. 

1 59. The computer system of claim 53, wherein said means for computing 

2 said computed network performance metric between said first network element and 

3 said second network element of said remaining network element pair fiarther 

4 comprises: 

5 means for creating a vector equivalent to said non-independent row of said 

6 first matrix by combining a plurality of said independent rows of said 

7 first matrix; and 

8 means for computing said computed network performance metric by 

9 combining a measured network performance metric of each network 

1 0 element pair of said second set of network element pairs corresponding 

1 1 to one of said plurality of said independent rows of said first matrix. 
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METHOD AND APPARATUS FOR ESTIMATING DELAY AND JITTER 
BETWEEN NETWORK ROUTERS 

Jain, Bijendra N. 
McCloghrie, Keith 

5 ABSTRACT 

A delay measurement technique according to an embodiment according to the 
present invention is based on the precept, ascertained by the inventors, that a link 
between network nodes will often contribute to the delay encountered between several 
different pairs of network nodes. Such a technique identifies the path between each 

1 0 pair of nodes by a list of links that form the path. Paths that are orthogonal are treated 
as being necessary for describing the delays encoimtered between nodes, and, once the 
requisite set of orthogonal paths has been derived, all other paths can be described in 
terms of one or more of these orthogonal paths. Such a technique also lends itself to 
matrix representation of the paths, and the use of matrix manipulation techniques in 

1 5 deriving delay and j itter. 
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